Counter overflows and clock drift bugs in aircraft and missile defense systems

A counter overflow bug happens when an unsigned integer variable storing a counter reaches its maximum value; when its already at the maximum, as soon as one more value gets added to it, the variable will reset back to 0 and continue counting up from there. If the rest of the software is not expecting this counter reset, it can result in system failures. So the longer the system is running for, the nearer the system will get to a counter overflow event.

Clock drift bugs happen when an internal timer/clock gets calculated incorrectly, causing it to slowly drift out of sync with real time. The longer the system is running, the more the clock drifts and the bigger the error gets. When real time clocks are out of sync, all sorts of downstream calculations can become inaccurate.

Both of these types of bugs typically have a workaround that requires the operator to reboot the system every X hours or days.

Both counter overflows and clock drift bugs can occur in production systems when system testing didn't include a "soak test" - where you keep the system running for a very long period, typically measured in days rather than hours. The failure to pick up the bug during system testing means that a production system gets deployed with a bug, and the only workaround becomes a regular reboot of the system.

The kicker - these type of bugs have been happening in safety critical systems for at least the past 28 years

One of the most publicized cases was the failure of a Patriot missile defense system in 1991, in which the system failed to track an incoming Scud missile, due to a drifting and out of sync clock. Tragically, this particular failure resulted in the death of 28 U.S. soldiers based at a U.S. barracks near to Dhahran, Saudi Arabia. The workaround for this bug was to reboot the system every 8 hours, but unfortunately this had not been communicated to the base in time to avoid the disaster.

Another well known case was the counter overflow in the Boeing 787 Dreamliner firmware (2015), which meant the aircraft had to be rebooted once every 248 days to avoid a counter overflow bug.

The most recent case was an internal timer bug in the Airbus A350 firmware (2019), which requires that the aircraft be rebooted once every 149 hours. 

Avoiding bugs in safety critical systems  

Avionics systems are among the most complex things ever built by man. There's one leading example of a successful avionics project which was built from scratch targeting high quality and zero defects. Its the Boeing 777 - Boeing's first fly-by-wire aircraft.
For the 777, Boeing decided early on to standardize on the Ada programming language, at a time when using C was the norm. Compilers for Ada had been certified as correct, and the language itself included several safety features, such as design by contract and extremely strong typing.

Ada itself came about as a way to standardize all of the programming languages in use by the U.S. Department of Defense - before Ada they were using some 450 different programming languages. Ada was originally designed to be used for embedded and real time systems.

Ronald Ostrowski, Boeing's director of Engineering, claimed that the Boeing 777 was the most tested airplane of its time. For more than 12 months before its maiden flight, Boeing tested the 777's avionics and flight-control systems continuously - 24hrs - 7days - in laboratories simulating millions of flights. 

The 777 first entered commercial service with United Airlines on 7th June, 1995. It has since received more orders than any other wide-body airliner. The 777 is one of Boeing's best-selling models; by 2018 it had become the most-produced Boeing wide-body jet, surpassing the legendary Boeing 747.

Further reading - Boeing flies on 99% Ada

Follow @dodgy_coder

Review of Shopify - from a developer's point of view

I setup this online shop over a couple of weekends - Chow Slow - Slow Feeder Dog Bowls. After launch I continued to make various improvements and optimisations to the presentation and content of the site. Since launch three months ago, the online shop has so far taken 72 orders.

The eCommerce SaaS product Shopify was used to build the site, on the Basic Shopify (monthly) plan.

Here below are my opinions on Shopify:

Shopify - Good points

• Many payment options - I chose the Shopify Payments option. It accepts Apple Pay and Google Pay when on a mobile browser for example. You can tell that Shopify have extensive experience with taking payments - everything is really polished.
• Test mode works well for putting through test credit card orders and seeing the customer experience end to end. Can also password protect the site while its under development.
• The free themes are good and can be customised for most online shops. I used the free "Minimal" theme and then customised the layout of the home page, added several pages and added a blog.
• Great optimisations and layouts which work well on mobile. More than 60% of the traffic for the site comes from a smartphone browser as opposed to a traditional desktop browser.
• Really useful "Timeline" admin feature on the customer and order admin pages - see all the interactions that have happened with a customer in a simple timeline display, with a timestamp shown on each interaction.
• Excellent "Conversion Summary" admin feature - see the site behaviour of your customer from when they first visited your site to when they bought something.
• Comprehensive "Fraud Analysis" admin feature - for orders paid by Credit Card - recommendation of whether the transaction is likely fraudulent or not: probability shown as low, medium, or high.
• Ability for customers to sign up with either their mobile phone number or email address. Shopify automatically sends communications to the customer using either mobile SMS or by email.
• Easy to add social media accounts, which then automatically appear on the site footer.
• Products are easy to configure and easy to change in terms of page title and content.
• Ability to edit raw HTML in most places.
• Easy to setup a custom domain, with TLS/SSL included (which just works).
• There are SEO optimisations and settings available at a page level. You can change the page title, meta description and URL.
• Integration with Google Analytics is excellent - the 'Advanced eCommerce' option should be turned on - it results in a data going straight into GA for your conversions and sales.
• Page load time is good - bulk of the home page loads typically in 2-3 seconds.
• A comprehensive API for writing custom integrations with other backend software and systems.

Shopify - Bad points

• Marketing integrations - Facebook Ads for example - difficult and error prone to setup. Facebook Ads never worked for me after many headaches trying to setup. The marketing integrations seem like a quick way to lose money fast.
• Inability to structure the Products, Pages and Blog sections to be within a custom sub-folders structure (as is typically recommended to silo a website for SEO purposes).
• Checkout page customization is limited to Shopify Plus users.
• Checkout script editor is limited to Shopify Plus, so any custom logic to apply discounts has to be done through plugins.
• API is rate limited - will be fine for most users however ERP systems which run via batch processing or via polling may have issues. You need to implement everything as event driven webhooks rather than by polling the Shopify API.
  

Shopify - Areas for improvement

• I'm not convinced of the need for many of the Shopify apps (plug-ins) that get automatically suggested. I have a feeling that they will result in bigger page sizes and slower load times. I have not needed to use any apps yet.
• Its sometimes hard to find the relevant place for a setting - whether under preferences of the online shop or under settings of the account itself. I should probably make more use of the search bar at the top for this in the future.

Shopify - Conclusion

I highly recommend Shopify for setting up an online shop. When launching, your focus will be taken up with gaining your first customers, online marketing and the delivery process, so having a solid full featured website behind you is crucial.



A bit more about our online shop - Chow Slow - Slow Feeder Dog Bowls

I setup the site for my wife, who runs the business day to day. She handles the processing of orders and deliveries. We came up with the idea of selling to this niche market after finding out about this type of product in 2018. Our pet beagle (Ripley) had an issue of regular regurgitation due to eating too quickly. The slow feeder bowl product solved his problem and results in more relaxed meal times every day.


Follow @dodgy_coder

Inactive/Failing Mobile Apps: The Many Benefits of Regular App Updates

This article applies mainly to mobile apps which are not in active development. It applies equally to mobile apps on the public app stores and also to enterprise mobile apps which get distributed via other means (such as MDM software).

Example scenario of an inactive/failing app:

• The active marketing of your app that happened at launch has now stopped
• Your app's last update was several months ago
• The uninstallation rate has started rising above your installation rate month on month

A way to prevent this happening is to release regular app updates, even when there are no major changes to your app's code.


Benefits of releasing regular app updates:

• Gain new users; many users look at the last release date on the app store page before downloading your app. Anything over a year old can be immediately rejected as out of date.
• Prevent uninstalls; seeing your app has been updated on their phone keeps it in the user's mindshare, so they know you are still actively developing it.
• Update the dependencies; you will likely have several dependencies on open source libraries and mobile SDKs. The more often you release, the more often these can be updated too.
• Update the toolchain; Xcode / Android Studio / Visual Studio get updated once every month or two. Same applies to hybrid and cross platform toolchains such as Unity, Ionic and Xamarin.
• Update the store listing; since you're doing a build, take the opportunity to review and tweak the store listing and screenshots. Apple's App Store Connect website and Android's Google Play Console get updated regularly with enhancements you can take advantage of.
• Less build issues; there will be a lower chance of a major build issue if you regularly update your toolchain and its dependencies. In my experience, the longer you wait between app builds and releases, the greater the chance of a major build issue the next time.
• Source control change visibility; the project is kept active in terms of source control history; especially relevant if its an open source project with other potential developers able to see the change history on github or bitbucket.
• Developer benefits; you don't end up with a legacy mobile app that noone can get building anymore.

What if there are no significant changes to your code?

• Have the dependencies or toolchain been updated? If yes, that means your app has gained some potential bug fixes and enhancements for free - just by doing a build.
• There's always some minor refactoring you can do to make small improvements to your codebase. Just spend a couple of hours if that's all you can spare.
• The release note; just use the standard "Bug fixes, performance and stability improvements" if you can't think of anything else.

How often to release?

• The more often the better.
• Aim for a regular release cycle of between 1-3 months.

Further reading

Distribution of Android apps per download range
https://www.statista.com/statistics/269884/android-app-downloads/

Download distribution of Android apps, and lifetime growth rates per download range
https://www.appbrain.com/stats/android-app-downloads

Statistics about the release schedule of apps on the Apple app store
https://stories.appbot.co/how-often-should-you-update-your-app-9405b85a967c

Best practices for Apple app store updates
https://developer.apple.com/app-store/app-updates/


Follow @dodgy_coder

Subscribe to posts via RSS