Posts on Iain McLaren

Linking agile to payments - what is the best way to pay for software development?

Sat, 14 Jun 2025 00:00:00 +0000

The agile methodology reflects how software is built. It’s complex and messy. So what is the best way to pay?

The agile manifesto does not mention when developers should be paid

I have been thinking about David Kellam’s comment about my last agile post:

Agile simply reflects the nature of software development, and specifically the nature of innovative software development.

I agree. The agile methodology simply reflects how software is built. When building something new, agile is a way to manage the complexity of an inherently changeable and fluid process. The focus needs to be “customer collaboration over contract negotiation”¹.

However, the agile manifesto does not mention when developers should be paid.

Let’s just throw cash at it

So what do you do if you have no software development expertise and want to pay a supplier to build software for you? The solution cannot be to throw an indeterminate amount of cash at the supplier until they (hopefully) build the product to your satisfaction.

In other words, agile is not an excuse.

Keep contract pricing incentives as simple as possible. But keep them.

Building a house is also a complex and messy process. As a customer², it is perfectly reasonable to ask a builder to quote a fixed price to build your house. The builder will ask for input as needed and change the price if the customer asks for something new or different. There is risk on the builder’s side, but the builder (not the customer) is the expert and controls and manages the project. The customer pays the builder to manage the risk.

Similarly, if you are a customer with no software development expertise and want to pay a supplier to build software for you, it is perfectly reasonable to ask for a fixed price. However, you will need to accept (like with the house) that your supplier will seek your input but you won’t get to micromanage how the house is built. If you want to change something then the price will go up.

Use agile. Or use waterfall. Or don’t. The customer won’t care as long as the software is delivered on time and on budget.

The customer usually doesn’t care how software is built

The customer doesn’t care how software is built. Just that it’s built well, relatively fast, and for a reasonable price.

The agile (i.e. software development) contracting risk if we don’t include waterfall pricing breakpoints (i.e. a pricing incentive to deliver on time and on budget) is the supplier iterating, at the customer’s cost, without producing much or any great production code. Removing this supplier risk means removing the supplier’s financial incentive to build fast and smart.

If you pay your supplier to build then it is reasonable to ask them to take on pricing risk with some sort of fixed price mechanism. Alternatively, if the project really needs constant agile customer input then the customer needs to carefully track and manage the project. The customer needs to be able to get out, or replace the supplier, if the project goes off the rails.

Sometimes avoiding the problem is the best solution

As a customer³, a better approach might be to just buy an existing product or build an in-house software development team. It is no wonder that many customers decide to just buy an existing completed solution, or build software development teams in-house to avoid this challenge.

1. agilemanifesto.org ↩︎
2. See what I did there scrum fanatics? ↩︎
3. Ah, the scrum stand-up memories. ↩︎

Drill down without slowing down

Wed, 11 Jun 2025 00:00:00 +0000

We can drill down without slowing down.

We are seeing increasing regulatory scrutiny of the information security of the financial services industry, including superannuation trustees and banks. The art is to drill down without slowing down.

https://www.apra.gov.au/for-action-information-security-obligations-and-critical-authentication-controls

https://www.oaic.gov.au/news/blog/outsourcing-cdr-obligations-the-buck-stops-with-you

Large technology suppliers that provide the same critical service to multiple companies often resist close scrutiny by their customers, such as the right to penetration test, conduct on-site audits, and otherwise monitor suppliers’ business continuity and security processes and systems. Managed poorly, customer intervention can increase costs, potentially slow development, and even create its own security risks. Audits by one customer should not allow that customer to access other customers’ data.

That being said, Australian regulators are clearly focused on ensuring that the financial services industry closely monitors and manages its suppliers. Customers need to be able to engage with suppliers, drill-down to confirm how suppliers operate, and validate that suppliers have systems and processes in place to keep technology systems and data secure.

There is an art to this when negotiating contracts between the supplier and the customer. Suppliers that do this best set up processes to provide the same form of access to all customers. These processes allow customers to drill down without slowing down.

Open source software licences

Thu, 29 May 2025 00:00:00 +0000

A very interesting study interviewing lawyers who work with open source software and the challenges they face:

https://arxiv.org/pdf/2403.14927

The key findings certainly track with my experience. Particularly how using off-the-shelf open source software licences, such as appropriate Open Source Initiative (OSI) licences, can drastically simplify compliance for both software developers and lawyers.

(article via James Oh @leafless@fosstodon.org)

Oh agile

Sun, 25 May 2025 00:00:00 +0000

Oh agile.

https://hwlebsworth.com.au/the-wide-world-of-agile-projects-lessons-we-can-learn-from-austech-applications-v-oz-wide/

It’s an evergreen theme. When your lawyer hands you a supplier’s services contract, is it clear and are you happy with what you will get, how much it will cost, when you will pay, and how much you will pay if you want to exit early?

Generally the contract payment process will either be fixed price (we pay when the thing is successfully delivered) or time and materials (we pay by the hour, day, or month and stop paying and using the service if we are not happy).

Agile, and agile contracting, is now old hat. I first wrote about how to draft and structure an agile contract a decade ago (!), and reading back I stand by what I said then:

https://iainmclaren.com/posts/2015-01-28-agile

Agile does not mean no project management or controls. Quite the opposite. An agile contract is a contract that provides just enough structure (and specifications) to ensure that both the supplier and customer act as the other expects. But an agile contract should then get out of the way of the individuals who actually deliver what the customer wants. That is the power of agile principle five:

Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.

I am always happy to help you with drafting and negotiating your contracts. I would also be very interested to hear your examples of agile contracting victories and defeats.

How to draft an AI contract

Thu, 26 Sep 2024 00:00:00 +0000

Best practice for using AI (i.e. in this context large language models) is in a state of flux. Our clients tell us that they see a lot of abstract high-level articles about AI, but little concrete practical advice about how to draft an AI contract.

What should we consider when drafting and negotiating an AI contract?

You will probably either download AI models directly from the model creator or access AI models via a service provided over the internet

Large language models come in two forms:

(a) models provided as a service that you can access remotely, most famously the OpenAI models (such as ChatGPT); and

(b) models that you can download and use on your own computers. For example:

Google DeepMind (Gemma-2-9B);
Meta (Llama 3.1-8B);
Microsoft (Phi-3-medium-128K-Instruct); and
Mistral AI (Mistral-Nemo-Base-2407).

Your AI contract will probably be a software as a service agreement and not a model licence agreement

Unless your company is building its AI models from scratch (i.e. unless your company is OpenAI, Google, Meta, or a similar organisation), to use AI your company will probably either:

procure AI models directly from a model creator under a model licence agreement; or
access AI models via a service provided over the internet by a supplier under a software as a service agreement.

If customers download the models (i.e. option (b) above), then their AI contract will be a model licence agreement governing how the customer uses the models downloaded to their computers. These customers will probably end up using the standard terms of use for the models provided by the applicable model creator listed above.

Alternatively, if customers use models accessed remotely (i.e option (a) above - such as ChatGPT), or customers use other AI vendors to process customer data and access AI functionality via the option (a) or (b) models on behalf of the customer, then the customer AI contract will be a software as a service agreement with the AI vendor.

Your AI contract will probably be with an AI vendor that is not the underlying model creator or provider

While typing prompts into a ChatGPT interface may be fun, many AI agreements signed by business customers are with AI vendors that do not create the underlying AI models. Instead, these AI vendors build industry-specific services for customers using AI models provided by model creators.

These AI vendors build their own software that interacts with AI models via the option (a) or (b) models discussed above, and these AI vendors host the software and customer data on the AI vendor’s servers. These AI vendors often host the customer’s information and access the models using Retrieval-augmented generation (RAG) and store the customer data in vector databases used to interact with the AI models.

In other words, customers upload data and documents to these AI vendor’s servers, the software on the AI vendor’s servers interacts with the AI models, and then the software on the AI vendor’s servers sends responses in the form of data and documents to their customers.

Key considerations when drafting an AI software as a service contract

(a) Your AI software as a service agreement should contain all the terms and conditions contained in a non-AI software as a service contract

For example, your AI contract should include the following details (mostly paraphrasing APRA requirements here):

details regarding the scope of the arrangement, services to be supplied, and fees;
service levels and performance requirements;
details regarding the form in which data is to be kept and clear provisions identifying ownership and control of data;
reporting requirements, including content and frequency of reporting;
audit and monitoring procedures;
business continuity management and force majeure processes;
terms and conditions in relation to confidentiality, privacy and the security of information (including whether the vendor complies with ISO 27001 or a similar security standard);
data location and transfer restrictions;
default arrangements and termination provisions;
dispute resolution arrangements;
liability and indemnity clauses, and obligations for the supplier to comply with applicable laws;
sub-contracting obligations;
insurance obligations; and
to the extent applicable, offshoring arrangements (including through sub-contracting).

(b) Data de-identification and aggregation (model creators will not necessarily keep customers’ information confidential)

Customers should assume that the model creators will use customer request information to improve their products (i.e. model creators will not necessarily keep customers’ information confidential). To avoid this potential problem, customers can either not use personal or confidential information when using these AI services, or can de-identify the information before it is provided to the model creator.

Customers should ensure that their software as a service agreements (whether AI or non-AI) require the AI vendor to keep the data that customers upload secure, comply with privacy laws, and only use the confidential and personal information of the customer in order to provide services to the customer.

In other words, if the goal is to not provide identifiable information to the model creator, customers can either:

de-identify the information before it is provided to the AI vendor; or
provide the information to the AI vendor under an AI contract with appropriate security, privacy, and data use limitations. The AI contract can then require the AI vendor to de-identify the customer information before providing that information to the model creator.

(c) Vector databases are commonly used for AI services, and may contain customer personal information and confidential information

Depending on the AI model used, the AI vendor may store customer information in a vector database and interact with the AI models using Retrieval-augmented generation (RAG). A vector database is a database that stores vectors (fixed-length lists of numbers which are mathematical representations of data in a high-dimensional space) along with other data items which allows the user to search the database with a query vector to retrieve the closest matching database records.

Databases more commonly used by software as a service providers (such as relational databases and NoSQL databases) generally store customer records in the database as text. On the other hand, vector databases store customer data as fixed-length lists of numbers, and AI vendors then use the information contained in the vector database with the AI models to generate AI model responses.

While vector databases may not contain raw text, these vector databases may still contain retrievable customer personal information and confidential information. Even if customers create their own vector database and provide the database (or access to it) to the AI vendor, then the AI vendor may still have access to customer personal information and confidential information that will need to be appropriately protected and secured.

(d) Using AI in high-risk settings and changes in law

Global AI regulation is in a state of flux.

For example, in Australia, the Department of Industry, Science and Resources has released a Proposals Paper for consultation on Introducing mandatory guardrails for AI in high-risk settings, alongside a Voluntary AI Safety Standard. The proposal is that organisations developing or deploying high-risk AI systems will be required to do the following:

Establish, implement and publish an accountability process including governance, internal capability and a strategy for regulatory compliance.
Establish and implement a risk management process to identify and mitigate risks.
Protect AI systems, and implement data governance measures to manage data quality and provenance.
Test AI models and systems to evaluate model performance and monitor the system once deployed.
Enable human control or intervention in an AI system to achieve meaningful human oversight.
Inform end-users regarding AI-enabled decisions, interactions with AI and AI-generated content.
Establish processes for people impacted by AI systems to challenge use or outcomes.
Be transparent with other organisations across the AI supply chain about data, models and systems to help them effectively address risks.
Keep and maintain records to allow third parties to assess compliance with guardrails.
Undertake conformity assessments to demonstrate and certify compliance with the guardrails.

While most AI regulations that will affect most customers may be abstract or years away, preferably your AI contracts should:

require AI vendors to comply with the obligations such as the above obligations, track the use of this information and compliance with these obligations, and provide the above information to the AI vendor’s customers; and
allow for changes in law, and require AI vendors to comply with new and updated laws.

(e) Hallucination

Enough has already been said about AI hallucination. Customers should ensure that their AI use cases involve human checks and intervention where appropriate.

(f) An exit plan

For all software as a service arrangements (whether AI or non-AI):

(a) customers should have an immediate exit plan if laws change, allowing the customer to move to a new or different service (if required); and

(b) preferably the applicable AI contract should include terms that requires the AI vendor to delete all of the customer’s data if requested by the customer, noting that (as discussed above) the model creator may be able to continue using any customer data provided to the model creator even if the AI vendor intermediary deletes all its copies of the customer’s data.

Why is an organisation the 'technology leader'? Boring technology foundations.

Thu, 19 Sep 2024 00:00:00 +0000

If there are three to five large players in a highly regulated market, and one is seen as the technology leader, it is often because that organisation has already invested in replacing their technology foundations with boring technology. We can see this across industries, including the energy, telecommunications, hospital, and financial services industries, as well as for government services.

Having boring technology foundations allows the organisation to more easily build new things such as software, software and software as a service applications, and even Large Language Model (LLM) applications, using their existing systems and data. These organisations don’t have to start all their technology projects from scratch or, even worse, try to build a seaweed of tangled technology on top of their existing decades-old technology foundations.

Which technology is boring technology? It’s an educated guess.

Boring technology is software and systems that are likely to keep working for decades. When we build or buy technology systems, all we can do is to take an educated guess as to whether the technology that we build on will be backwards compatible (i.e. will not break our existing systems), will not disappear, and will only change in a way that gives us plenty of time to update our existing systems.

For example, we can make educated guesses about what technology is boring, and will stay boring for at least a decade, by considering the following:

(a) Do lots of people use the technology? The greater number of companies that use the technology, the less likely it will be that the technology will disappear overnight. If many companies use the technology, then those companies will be willing to pay vendors and third-party service providers to continue supporting and maintaining the technology.

(b) Is the technology open source, commodified, or does it at least use open data standards? Any technology which is controlled by a single company can be a risk. Whether it is a software vendor that does not release its source code, or a software as a service provider that locks you into their platform in a way that makes it difficult or impossible to extract your data.

(c) Do the developers of the technology respect existing users when they change the technology? Microsoft (Windows) and Apple (macOS), for example, have a long history of going out of their way to try to ensure that software that runs on old versions of their operating systems continues to run on the latest versions of their operating systems.

What are some examples of boring technology?

It is becoming increasingly rare for large organisations to build their software from scratch. For example, building your own enterprise resource planning, customer relationship management platform, or core financial management platform, from scratch rarely makes sense. Instead, organisations seen as technology leaders often pay for an industry-specific, relatively boring, third-party “core platform” and then use, build, and innovate, using that platform.

At a low level, commentators also consider technologies such as Linux distributions, Postgres databases, the Go programming language, and the Amazon Simple Storage Service (S3) protocol, to be boring.

All of these low-level technologies are widely used and are either open source (Postgres and Go) and/or are provided by multiple vendors allowing users to easily move from one vendor to another (Linux and S3).

A personal anecdote (I just had to update my boring technology after 7 to 10 years … and I’m OK with it)

I have been thinking about this during the last few weeks because I recently spent a bit of time doing technology work to effectively stay in place. I needed to move:

from version 1 to version 2 of the Amazon Web Services (AWS) s3 Go system development kit library https://aws.amazon.com/blogs/developer/announcing-end-of-support-for-aws-sdk-for-go-v1-on-july-31-2025/; and
from version 1 to version 2 of the 1Password command line interface https://app-updates.agilebits.com/product_history/CLI.

Why was I OK with this? Version 1 of the Amazon S3 SDK library was supported for about 10 years, and version 1 of the 1Password command line interface was supported for about 7 years. Both also have excellent migration guides making updating the code from version 1 to version 2 relatively simple and not particularly time consuming.

I won’t say it was a joy, but it was relatively painless.

Who remembers Y2K? That was a boring technology problem.

If, like me, you are old enough to remember Y2K, you will recall that Y2K happened because people who built software in the 70s, 80s, and early 90s did not expect that their software would be used for 10, 20, or 30 years (i.e. into the 2000s).

It is a technology cliche that any software that you throw together quickly to fix a problem in a hacky way is bound to stick around. Effectively forever.

It is inevitable that the technology infrastructure that we build on will change out from under us

Similarly, it is quite a shock for software engineers the first time that they build a technology system, and the underlying libraries, databases, operating systems, or other technology, used to run the system suddenly causes their software to break because the underlying technology changes dramatically or disappears completely.

The vendor stops developing the technology. Or the operating system provider creates a whole new operating system that is incompatible with the software engineer’s software, and all of the software engineer’s company’s users move to the new operating system.

When software engineers praise boring technology, this is what they are talking about. Even though we know that there is no stopping technology from changing over time, nor would we want it to.

And only having to make simple changes to existing systems once every 7 to 10 years?

For me, that is as good as it gets.

Have you checked your software stored in escrow lately?

Thu, 12 Sep 2024 00:00:00 +0000

Have you checked your software stored in escrow lately?

https://www.mixonline.com/business/inside-iron-mountain-its-time-to-talk-about-hard-drives

For critical software and software as a service (SaaS) projects, large customers often require their suppliers to store the source code for the supplier’s software in escrow. This allows customers to access this source code, and hopefully fix bugs in the software, even if the supplier goes insolvent or otherwise stops providing the service.

Iron Mountain is often used as an escrow agent, but there is a potential problem:

A few years ago, archiving specialist Iron Mountain Media and Archive Services did a survey of its vaults and discovered an alarming trend: Of the thousands and thousands of archived hard disk drives from the 1990s that clients ask the company to work on, around one-fifth are unreadable.

To be clear, this is not Iron Mountain’s fault. It is up to the customer (i.e. the software supplier) to decide how to store data (i.e. the software source code) in escrow. There is a vast difference between uploading the software source code to a service like Amazon’s Simple Storage Service (S3) which is built to minimise data loss over time vs just delivering a physical hard drive to a trusted third party to store for decades. Hard drives all fail over time.

It’s a good reminder of the usefulness of annual business continuity tests.

When negotiating contracts for our clients, we often find ourselves helping our clients think through how to ensure that their business continuity plans will work in practice during the first few years and potentially decades into the future.

What happened with YubiKeys and why it is interesting (but a bit of a non-event) from a security perspective

Wed, 04 Sep 2024 00:00:00 +0000

https://arstechnica.com/security/2024/09/yubikeys-are-vulnerable-to-cloning-attacks-thanks-to-newly-discovered-side-channel/

1. What are YubiKeys and what are they used for?

YubiKeys are little hardware dongles (see the picture in the article below) that are commonly used for two-factor authentication. They are built based on the FIDO (Fast IDentity Online) standard.

YubiKeys are commonly used like house keys, but for computers. Secure laptops and technology systems can be set up in such a way that users can only access these systems if the user has plugged in their YubiKey.

2. What is the problem: YubiKeys can be copied.

Regular house keys can be copied if we take them to a locksmith. YubiKeys are designed so they cannot be copied even if the attacker has physical access to the YubiKey.

Like locksmiths can clone house keys, security researchers have discovered a way of cloning YubiKeys.

3. What is the solution: Replace the YubiKeys.

This appears to be a problem with the ‘firmware’ or built-in software on the YubiKey. For security reasons, YubiKeys are designed in a way that do not allow this firmware to be updated.

The only solution is to replace the YubiKey with a new YubiKey with the latest firmware that does not have this problem, and securely dispose of old YubiKeys.

4. Is this a major problem if our company uses YubiKeys?

Probably not. As the article points out, actually exploiting this problem requires physical access to the YubiKey, and seems to be very difficult and expensive in practice:

“The attacks require about $11,000 worth of equipment and a sophisticated understanding of electrical and cryptographic engineering. The difficulty of the attack means it would likely be carried out only by nation-states or other entities with comparable resources and then only in highly targeted scenarios. The likelihood of such an attack being used widely in the wild is extremely low.”

Very few people and organisations are targets of dedicated nation-state actors willing to invest this amount of time and resources.

As always happens when issues like this are identified, security researchers will try to find ways of exploiting this problem in simpler and cheaper ways. In the meantime, YubiKey users can replace their YubiKeys and securely dispose of their old YubiKeys.

That time when I spent two weeks hunting down one software bug

Tue, 27 Aug 2024 00:00:00 +0000

This post is a response to two questions that I am commonly asked. ‘Is coding fun?’ and ‘What is coding like?’

That time when I spent two weeks hunting down one software bug.

TV shows about computer programmers are nothing like being a coder

TV shows about any type of computer programmers, particularly ‘hackers’, often show teenagers wearing hoodies in dark rooms typing at 3000 words per minute and saying “I’m in” like Neo from the Matrix.

The reality is that a lot of computer programming, and bug or exploit hunting, involves the coder sitting in front of a computer reviewing code for hours and then saying … ‘Hold on, that doesn’t work’. Again, not great television.

Once you start coding it is hard to stop

I learned early on in my career that building software that is used by other people is addictive. Joy comes from creating something that makes the lives of other people easier. Once you start it is hard to stop.

Coding is, by its nature, a solitary activity. It involves solving hard problems in a way that is difficult to describe and not necessarily interesting even for other coders.

Bug hunting can be frustrating … but fun

I recently spent two weeks hunting down an elusive bug. The process was frustrating but ultimately fun. I am writing this because I think that the details might be interesting even for people who are not particularly interested in technology.

My software had a bug

For context, I have built and run software that works like Google Drive or Dropbox. The software stores files in the cloud. The files are automatically synchronised between multiple Apple computers.

There is also an iPhone and iPad app on the Apple App store which allows users to view and interact with the uploaded files.

The bug … and why I was fooled

I went on a trip for two weeks.

When syncing the files, the file sync service runs a full check of all downloaded files and then compares (amongst other things) statistics like the number of files, folders, and total stored sizes of files on both the laptop and in the cloud. These numbers are always changing as different users change the files and folders on their laptops.

If these statistics on the server and the laptop match, then all files are synced, and the software can stop checking files and folders. If not, then the software on the laptop runs another full check of all downloaded files and keeps doing so until these numbers do match.

The problem was that the laptops were running this full check, and these statistics on the server and laptops never matched. The software never stopped churning through all of the files and folders to check for changes even when the users were not changing any files or folders.

The journey

1. Bugs in this software tend to be scale issues or edge cases

Writing the basic code to synchronise files between a computer and the cloud is relatively simple. The tricky parts are:

scale issues, or ensuring that the software works for millions of files, with a collective size of multiple terabytes, for large numbers of users for a price that is cheaper than Apple iCloud / Google Drive / Dropbox; and
dealing with edge cases.

2. The edge cases for this software are generally file edge cases, and internet/network edge cases

(a) File edge cases

File edge cases include dealing with what happens if:

two or more files with the same name but different contents are uploaded at the same time;
files and folder names use characters that are problematic when used on Apple laptops (e.g. string normalisation);
a folder is deleted on one computer and a file on another computer is updated at the same time; and
file attributes (such as making the file read only) for the same file are different on two or more different computers;
etc.

(b) Internet/network edge cases

Networking edge cases generally involve dealing with flaky internet. For example, we need to deal with what happens if:

the internet is not available for one or more computers for seconds, hours, days, or weeks; and
internet that is available but flaky in weird ways.

3. Wait … how can internet access be flaky in weird ways?

I am glad (the hypothetical) you asked. Wi-Fi, particularly hotel Wi-Fi, is often bad in strange ways. For example some Wi-Fi services:

stop downloading files after an arbitrary number of megabytes of data, or after an arbitrary time period;
stop uploading files after uploading an arbitrary number of megabytes of data, or after an arbitrary time period; and/or
randomly get stuck and pause uploads and/or downloads forever.

Travelling is great because it helps test for these types of problems.

4. Ah, so our bug has to be a weird Wi-Fi issue right?

To find a new bug that we have not encountered in the past, we generally look at what has changed. In this case I was accessing the internet using potentially flaky Wi-Fi. This is not the first time that I have had this problem while travelling and had to update the software to cater for various bad types of Wi-Fi. It is how I relax while on holiday!

But I couldn’t find the problem. The laptop checked all of the files and folders without reporting any errors. The server was processing these folders without any errors. However, the statistics (e.g. the total number and size of files and folders) on the laptop and server were never the same.

I even tried creating a new test account with the same files and folders, and uploaded all of the folders to the cloud server again, in case this was a strange database corruption issue. The numbers still did not match on the server and laptops.

I was stumped.

5. Hmm. Still not magically fixed after I returned home

I was unable to fix the issue while travelling and, interestingly, the problem did not go away when I returned home to reliable internet access. Well, maybe saying that any home internet in Australia is ‘reliable’ is too strong. Let’s just say that my home internet access is unreliable in a predictable way?

Anyway, this did not seem to be an internet/network edge case issue.

6. Time to use brute force. This is where I gave up and manually checked the files and folders to diagnose the issue

In the end I gave up. I ran a brute force comparison between all of the files and folders on the server and one laptop and compared which files and folders were consistently on the server and not on the laptop (and vice versa). For context, this software stores millions of files.

In the end, I tracked this problem down to a folder containing the following files (I have changed the real name of the person to ‘Alex Correia’ here):

“/path/to/17635-Alex Correia.pdf”
“/path/to/17635-Alex Correia.PDF”
“/path/to/17635-Alex Correia.RTF”
“/path/to/18340-Alex Correia.RTF”
“/path/to/23073-Alex Correia.RTF”

I looked at these file names for about 20 seconds before it clicked. The problem is with the names of the first two files on this list.

The problem

Apple laptops work using filesystems that are case insensitive. This means that all of the following different file names are the same file:

“/path/to/ABC.DOC”;
“/path/to/ABC.doc”;
“/path/to/aBc.doc”;
“/path/to/abc.doc”; and
“/path/to/abc.DOC”.

These cannot be five different files.

I knew this. However, this had not been a problem for the sync software in the past because there will never be a file on any one laptop that is named “/path/to/abc.DOC” at the same time as another file exists on the same laptop named “/path/to/abc.doc”. Therefore, the software on one laptop will not try to upload both “/path/to/abc.DOC” and “/path/to/abc.doc”.

What seems to have happened is that one laptop uploaded “/path/to/17635-Alex Correia.pdf” to the server and another laptop uploaded “/path/to/17635-Alex Correia.PDF” to the server at the same time, so the server recorded entries for both files. Each laptop then checked whether both “/path/to/17635-Alex Correia.pdf” and “/path/to/17635-Alex Correia.PDF” (i.e. two files) were stored on the laptop. The laptop operating system indicated that yes, both “/path/to/17635-Alex Correia.pdf” and “/path/to/17635-Alex Correia.PDF” were indeed stored on each laptop.

In other words, the server indicated that all files had been uploaded without issue, the laptops indicated that all files had been downloaded without issue, but the software insisted that there was one more file stored on the server than had been downloaded to each laptop.

Classic edge case. Classic race condition.

The solution … which took about 15 minutes

Once I found the problem, the fix took about 15 minutes.

I just:

updated the server software to always treat files with the same case insensitive file name (e.g. “/path/to/17635-Alex Correia.pdf” and “/path/to/17635-Alex Correia.PDF”) as the same file; and
built a mechanism to deal with what happens if “/path/to/17635-Alex Correia.pdf” and “/path/to/17635-Alex Correia.PDF” are uploaded from two or more different laptops at the same time.

And everything started working again.

Golang database locker (calmdocs/dblocker)

Tue, 20 Aug 2024 00:00:00 +0000

How I access databases in golang:

github.com/calmdocs/dblocker

Golang database locker. A simple library to lock a shared database session for each “user” or “id” behind what is effectively a RWMutex.

The 20% cut - Tech cost cutting trends

Tue, 20 Aug 2024 00:00:00 +0000

The 20% cut: Tech cost cutting trends

The background: The tide is going out

For companies that are not building underlying generative AI models and hardware, we can see a quick shift from “Oh look at the amazing AI product that you are using.” to “We are hitting a downturn and I need you to take 20% out of your cost base within 6 to 12 months.”

AI is a bubble. A load-bearing bubble, but a bubble. Don’t build a house on it if you can avoid it. Even if your business is another AI grift, you probably only need to import openai. ― Ethan McCue

The task: The economy may ebb and flow, but the cost cutting target always seems to be 20%

A large number of business transformation, change, business process outsourcing, and technology projects are implemented because a senior executive is instructed by their manager to reduce the costs in their part of the organisation.

The executive engages management consultants who analyse how their part of the business is run. The consultants come up with a plan that reduces costs by about 20%. It always seems to be about 20%.

The general consensus seems to be that this happens because 20% is a ‘safe’ number. All transformation projects involve expense and risk and are usually not with pursuing if the savings will be lower than 20%. And any higher number may risk seriously damaging the business.

The process: Technology has changed over time but the process to cut costs by 20% … has not

Stripped down to basics, companies seem to approach the 20% cut in the same way that they did so 20 years ago.

Companies look at the following:

(People and processes) What are their people costs, and how much of what the company does is insourced (i.e. employees) and how much is outsourced (i.e. to contractors and third-party suppliers).
(Technology) What are their technology costs, and how much of this is insourced (i.e. run by employees) and how much is outsourced (i.e. run by contractors and third party suppliers).
(Pain points) What are their biggest pain points? In other words, which functions within the organisation are inefficient and expensive relative to their peers and switching from insourcing to outsourcing or vice versa.

The past: Technology used to be local

In the olden days (i.e. the 90s) companies either ran their own computer systems or hired outsourcing providers to do so. There was no hybrid model.

If we wanted to outsource our finance function, for example, we either required an outsource provider to run the outsourced technology as well as the service, or required all of the staff of the outsourcing provider to work at our premises.

The internet wasn’t fast of stable enough to run critical computer systems remotely.

The present: Technology is now global

This is now changed. Technology is now truly global.

Transformation projects are now run based on the assumption that knowledge workers can work from anywhere. The internet is fast enough to not require most knowledge workers to work together in offices.

The current and future trends: Boring is best

We can see the following trends in cost cutting transformation projects:

(Using expert Software as a Service providers) The continuing move from running software in-house to using expert software as a service (SaaS) providers of HR, finance, and other systems. (Using global hosting providers) The continuing move from hosting computers on-premise to using global hosting providers like Microsoft, Google Cloud Platform, and Amazon Web Services.
(The move away from lift and shift) Many transformation programs still involve replacing employee teams with an outsourced service provider, with the outsourced provider’s employees doing exactly the same work as the existing team in exactly the same way. However, we are seeing a continuing move away from this approach.
(The move towards remote access) Knowledge workers can now work remotely. While we complain about internet speeds in Australia, internet access is now fast and stable enough for most work performed by most knowledge workers.
(The move towards burning it all down and starting again from scratch) Best practice is now to pay an expert outsourcing provider to run the service (HR, finance, technology, etc.) in the same way that these providers do for all of their other customers. Using a common technology platform means that all customers benefit from a best of breed service that is continually updated by the supplier in a way that shares costs amongst all customers.
(A slow move to flashy (e.g. AI) but not to cut costs, at least not yet) AI may eventually become the leading way to cut costs but we are not there yet.
(The move towards boring) In the meantime, customers still seem to be focused on their people and processes, their technology, and their pain points. It may not be as exciting as just starting a new AI project, but this is still where costs can be saved.

Why cyber insurance is becoming like health insurance

Mon, 12 Aug 2024 00:00:00 +0000

The health insurance industry has a significant influence on how health care is provided. Similarly, cyber insurers may increasingly influence how organisations run their technology systems and secure their data.

Data breaches are still relatively rare

A whole industry has recently been built around mitigating and managing the risk of data breaches.

There used to be much less of a focus on cyber insurance and the risk of data breaches. Data breaches were rare, and the insurance premiums and payouts were relatively low. Contracts that customers signed with suppliers also reflected this. For example, customers strongly resisted including any liability cap limits for confidentiality and data breaches in their contracts with these suppliers.

That being said, data breaches are still relatively rare. A data breach is a classic low probability high impact risk.

Large organisations often default to storing as much information as possible

Before data breaches became more common, there was no real downside to organisations storing as much information as possible.

Even if organisations cannot think of a use for the data that they currently store, they know that they might come up with a profitable use-case in the future. Particularly as machine learning models are improved by using large quantities of useful data.

Historically, there was relatively little incentive for organisations to limit the amount of data that they store, or spend a lot of time and money on security

There is still a strong push within organisations to retain as much data as possible. For example, organisations are starting to recognise that applying machine learning models to their large data stores can lead to profitable results.

For the individuals who work in organisations, there is often little real incentive to limit the amount of data that they store on behalf of these organisation. Even if there is a data breach, the focus of organisations is often on increasing security as opposed to limiting the data that they store.

Speaking of day-to-day incentives, in large organisations, the security team that responsible for preventing data breaches is also often completely separate from the parts of the organisation that are responsible from profiting from large troves of data.

With the best will in the world, it is also unlikely that data breach prevention will ever be the top priority of boards and CEOs. Data breaches are low probability but high impact risks, and it is probably not realistic to expect that boards of most organisation will be stacked with directors with strong technology experience and expertise who understand how to appropriately manage these risks.

However, customers are quickly learning that their data stores can be toxic

However, customers are quickly learning that their data stores can be toxic.

For example, the European Union led the way with the General Data Protection Regulation (GDPR) by imposing enormous fines, linked to percentages of revenue, on organisations that do not appropriately store and use personal information.

Australia has followed suit with similarly large penalties under the Privacy Act 1988 (Cth).

As a result, cyber insurance is becoming prohibitively expensive

Put simply, all insurers calculate the probability of a potential payout and the amount of the potential payout and charge their clients accordingly.

When there were no cyber insurance payouts, or relatively low payouts, the cyber insurance premiums were also low. Cyber insurance payouts are still relatively rare, but insurance premiums have become prohibitively expensive as data breaches have become more common and the potential payouts larger.

Short of government intervention, the only way to reduce cyber insurance premiums is to reduce the risk to cyber insurers

To date, most commentary in relation to cyber insurance seems to be based on the assumption that the risk of data breaches cannot be reduced in a meaningful way. The assumption is that organisations will generally just continue to run their technology systems without any meaningful external controls, limitations, or mandatory risk mitigation strategies.

To give Australian Governments credit, there is a clear legislative focus on ensuring that the technology systems of critical organisations such as banks and other APRA regulated industries, hospitals, and other critical infrastructure, are run in a secure way. However, this has not stopped data breaches from occurring. There is only so much that governments can do to change the behaviour of organisations when the actions of the organisations are not directly driven by financial incentives.

Like with health insurance, the only way to reduce the risk to cyber insurers is to reduce the probability and/or amount of the payouts

Short of government intervention, there are two ways that cyber insurance payouts can be reduced:

offer toothless insurance; or
impose security obligations on organisations in exchange for discounted insurance.

Option 1: Like with health insurance, offer ’toothless’ insurance

Cyber insurers are willing to offer ’toothless’ insurance. Premiums could be reduced if the insurers don’t cover the largest expenses like fines, customer payouts, or company security remediation costs.

This is like buying the cheapest Australian health insurance that only covers inexpensive ’extras’ but doesn’t cover potentially huge expenses like the cost of extended private hospital stays.

However, large organisations have increasingly become wise to this risk. They have dedicated internal insurance teams who ensure that their organisation buys ‘real’ insurance to cover the cost of the real risks to organisations if there is a data breach.

Option 2: Like with health insurance, impose risk mitigation obligations on organisations in exchange for discounted insurance

The other option is for insurers to impose risk mitigation (i.e. security) obligations on organisations in exchange for reduced premiums.

Health insurers have already travelled down this path. To manage risk and premiums, health insurers work within a very complex regulatory regime and are deeply embedded in how hospitals and other health care providers provide services to patients.

On the other hand, traditionally cyber insurers have not had any real input in relation to how insured organisations run their technology systems, protect their data, and manage risk.

Cyber insurers often have relationships with security experts who step-in, as part of the insurance payout, to assist when data breaches occur. However, only stepping in after the data breach occurs is a deeply reactive response. Stepping in afterwards can mitigate the result of a breach to a certain extent, but only treats the symptoms of the breach rather than the underlying cause, which can be having insecure systems or otherwise not appropriately managing these risks.

Without treating the cause of the problem, organisations may still incur significant costs even if the best security team in the world is dropped in to help after the breach has occurred.

The problem of scale: What should cyber insurers require organisations to do?

This is a problem of scale. Even if they wanted to, there is probably no way that cyber insurers can actively manage or monitor the technology operations of all of the organisations that they insure. Which leaves cyber insurers with two options in exchange for discounted premiums:

impose security standards on organisations; and/or
stop most organisations from running their own technology systems.

Option 1: Impose security standards on organisations

There are a number of internationally recognised security standards that insurers could impose on organisations.

For example, in Australia, most of the security obligations that regulators impose on banks, hospitals, infrastructure providers, and other critical industries are based on the internationally recognised ISO 27001 security standard. Organisations can even be independently certified as being ISO 27001 compliant. However, for most smaller organisations, cyber insurers are unlikely to be able to push for such formal compliance because requiring such compliance will be too expensive and time consuming for the organisation relative to the risk of a data breach occurring.

Option 2: Charge a risk premium if organisations run their own technology systems

The second option is simpler. Only provide cheaper cyber insurance premiums to organisations that do not run their own technology systems.

This trend has already started in schools that issue their students, or require students to buy, iPads, Chromebooks, and other locked down devices. These schools effectively outsource managing security to the device manufacturers and service providers that run software in the cloud.

The reality is that very few organisations need to create or even run their own software. Most organisations just use the Microsoft office suite including Office 365 for document management and email, and standard finance and HR systems. Now that we have (relatively) cheap and fast internet access, all of this software can be run better, and more securely, by third parties in the cloud.

Even for large organisations, using Office 365 instead of running their own email systems is now the norm, as is organisations using global hosting providers such as Google Cloud Platform, Microsoft Azure, and Amazon Web Services instead of running their own data centers.

To get reasonably priced cyber insurance, eventually most customers may not run their own technology systems

The trend seems to be that most organisations may eventually come to the conclusion that it is too risky to run their own technology systems, particularly if insurers effectively charge organisations a premium to do so.

Most organisations may come to the conclusion that it is safer and cheaper to rely on global technology organisations (like Microsoft) to operate and manage technology services that most large organisations use (like email) rather than than running these technology systems themselves.

To manage risk and limit premiums, cyber insurance may increasingly become like health insurance

This is similar to what happened with health insurance. To manage risk and limit premiums, the health insurance industry now has a significant influence on how health care is provided.

Cyber insurers may soon have a similar influence on organisations particularly if organisations continue to insist on running their own technology systems. This may increase the stampede of large organisations following the crowd and outsourcing to global technology organisations that run software in the cloud.

Avoiding the business continuity trap

Mon, 05 Aug 2024 00:00:00 +0000

Real business continuity for critical industries like banks, hospitals, and government organisations

The challenge: Are we really ready for disaster?

We do a lot of work to assist banks and other financial service providers, hospitals, and government services organisations. To operate, these critical organisations need to have plans in place to deal with disasters.

These organisations rely on the services of their most critical suppliers. If the organisation does not have a robust continuity plan in place it can be catastrophic if those suppliers stop providing their services for any reason, including because of natural disasters, wars, if the supplier becomes insolvent, or (and I have seen this) the supplier just decides to stop providing the service.

The symptom: Leading with business continuity tools instead of a business continuity solution

There are a number of tools that I see customers commonly throw at this problem, such as (for technology suppliers):

building a ‘step-in’ process into supplier agreements that allows the customer to take over the management of a critical service provided by a supplier;
storing the source code of the supplier’s software in escrow so the customer can (in theory) fix bugs if there is a major problem with the supplier’s software; and
requiring the supplier to provide functionality allowing the customer to download all of its data from the supplier at any time, including when the customer wants to stop using the supplier.

The challenge is that, for most customers, the above may not allow the customer to get back up and running quickly and cost-effectively.

The trap: Most customers cannot take over and run supplier services themselves

The main reason why organisations use third party service providers for critical services is that they cannot provide or perform these services themselves. For example, some of the best technology security people I know work in hospitals. These people would never try to run a critical data centre themselves. They don’t have the expertise. That’s why they pay the suppliers.

In practice, this means that:

other than for the most sophisticated organisations, having access to the source code of the software doesn’t help;
if the supplier is insolvent or just stops providing the service that we use, there is nothing useful to ‘step in’ and take over; and
if the customer has not downloaded the latest copy of the customer’s critical data before the supplier’s servers are switched off, then this data may be lost forever.

The solution: Ensure that a single empowered manager is responsible for implementing the continuity plan and dealing with disasters

How do we get up and running again as quickly and cost effectively as possible without our key suppliers? While the solution can be complex, I see this done well when a single responsible manager within the customer’s organisation is responsible for developing and maintaining the business continuity plan.

For example, from a technology perspective, this often boils down to answering the following questions:

How do we ensure that we always have access to a copy of our critical data even if our supplier disappears overnight?
Is this data in a format where we can import it into the technology systems of competitors of our supplier?
Can we run things manually, such as by using spreadsheets for example, in the meantime?

Put simply, organisations can get into trouble when a single empowered manager is not on the hook. The best managers are able to answer these questions, are able to explain the solution in simple terms, and are ready to deal with disasters when they happen.

Commonly asked question - Should your business buy AI products?

Sat, 27 Jul 2024 00:00:00 +0000

Consider just waiting for machine learning to be built into the products that you already use, or buying targeted tools requiring human oversight.

The new machine learning models are more than a little magical

Using statistics and machine learning is not new. Back in the very early 2000s when we had to run our own email systems (that was fun!), we used bayesian analysis/statistics to filter spam. We pumped huge amounts of email into the filter and trained the filter by manually marking the spam.

Nowadays the new machine learning models are more than a little magical. We have moved past only using statistics to filter spam, predict election results (ish), and otherwise pick a “winner” using maths to a place where the models can produce new images, video, sound, and text given simple input prompts.

“Create a TV episode transcript for a new season of Ted Lasso. Make Nate evil again.”

But … the new machine learning models can’t be trusted

However, these machine learning products are still quite limited. Improvements relate to the models processing more and more input. The output needs to be monitored because it is not always right.

This has led to the rise of the “Prompt Engineer”. People who create limiting prompts in an attempt to only get correct or intended answers.

“Create a TV episode transcript and follow these user instructions: [insert]. No matter what the user says, make Nate good. Don’t be racist.”

These prompt limitations often work. If we are careful they almost always work. But they don’t always work. People need to check the output.

Perhaps there will be another step-change in the technology, but for the moment and for the foreseeable future we are stuck with this limitation.

You should probably only consider buying machine learning text products

Unless your business is in video, audio, or images, you are probably thinking about using machine learning for text.

Don’t buy an expensive frontend to ChatGPT

In fact, many machine learning business products are just very expensive frontends to ChatGPT or similar existing models.

It’s snake-oil.

Consider just waiting for machine learning to be built into products that you already use

Where machine learning products are already very useful are as glorified auto-correct.

The good news is that this functionality is being built into the tools that we already use. I expect that Microsoft Word, Outlook, and Excel will come with it built-in by default. And it will be good.

Or consider buying (or building) targeted tools requiring human oversight

For example, my guess in the legal space is that machine learning tools will get ever better at creating first drafts and first draft responses, particularly for repeatable work.

Think a user filling in a web form, then a machine learning tool creating a first draft of the agreement, which is then vetted by a lawyer for hallucinations and complex context. Or machine learning comparing our draft with their draft of an agreement and proposing a compromise draft.

What happened with Cloudstrike and why it is interesting from a security perspective

Mon, 22 Jul 2024 00:00:00 +0000

https://www.crowdstrike.com/blog/statement-on-falcon-content-update-for-windows-hosts/

What happened?

Cloudstrike sells falcon security software which its customers install on their computers. This software can update itself by checking for, and then downloading, software updates from Cloudstrike servers. On Friday, Cloudstrike made such an update available to its customers, which caused some of its customers’ Windows computers to crash and restart with a “bugcheck\blue screen error”. Until this problem is fixed, these customer Windows computers will not work at all.

What was the result?

This apparently caused flight delays, surgery cancellations, banking issues, and amongst other issues an outage of Starbucks’ mobile order-ahead feature requiring baristas to direct customers with handwritten signs to “plz come to walk-up.”

Was customer data accessed in an unauthorised way, lost, stolen, or held for ransom?

It does not seem so.

How do customer security teams generally react to problems like this?

Generally, major technology security problems are categorised as (1) business continuity / disaster recovery events which stop customers from accessing or using technology, such as third-party technology systems not being available, pandemics, power failures, telecommunications failures, war, insurrection, etc. and/or (2) a major security issue which often manifests as data breaches, viruses/ransomware/malware, hacking, social engineering (tricking people), denial of service attacks, etc.

How customer security teams are reacting

For impacted customers, this is definitely (1) a business continuity / disaster recovery event. While seemingly not malicious, any incident involving downloading third party software to multiple local computers which unexpectedly causes those computers to crash and not recover without intervention will also be treated by security teams as (2) a major security issue.

What makes this interesting from a security perspective?

Security specialists generally think about systemic technology risk in the context of the risk of very large numbers of customers using the same technology hosting provider (such as Google Cloud Platform, Amazon Web Services, etc.) or the same operating system (Microsoft Windows). This is interesting because one update, to one type of software installed on customers’ computers, provided by one software vendor has caused such significant issues.

Why Go is my favourite programming language

Mon, 08 Apr 2024 00:00:00 +0000

go is my favourite programming language.

Introduction (and why not rust?)

Every now and then I am asked why go is my favourite programming language. There are so many reasons, so here is a summary. Like all programming languages that I have used, there are also some issues with go and dark corners of go. To hopefully show that I am not too narrow minded, I list some of these issues and dark corners at the end of this post.

In a different timeline, this could have been an article about rust. Rust seems like a great language, but rust was released in 2015 which was after I first needed to build a tool using a programming language like go or rust. Rust seems to solve similar problems as go, and seems even more pedantic and opinionated than go … which is a compliment. But … well … I’m used to go. Maybe I will change my mind in the future?

In any event, I like my technology as old, boring, and universally available as possible (i.e. available using all the main cloud providers). For example, I like:

Debian or Ubuntu linux servers;
s3 compatible services for storage;
postgresql; and
golang, python, and bash scripts.

I don’t want to use tech that is likely to change in a way that breaks anything, or disappear in the near future. Rust is pretty old and boring, but not quite as old as boring as go.

Some reasons why Go is my favourite language

Go is fast

Go became my favourite programming language the second time that I had to switch from python to go for a project because python ran too slowly.

I wanted a language that runs close to the speed of c, but with safeguards against my bad coding such as enforcement of strict typing and memory management.

Go is pretty, simple, and easy to understand

I like programming languages that are simple, graceful, and easy to read and understand.

In my eyes, uglier languages include:

c and c++, although they are forgiven because their ugliness (particularly lack of memory management) bring us closer to the machine without requiring us to resort to assembly.
c#, java, and objective c. Urgh, ugly, ugly, ugly.

Middling languages include:

rust. I like a lot about Rust. My only very minor gripe is that I find code with double colons “::” a bit hard to grok. I will probably get over this given time.
python and ruby. Python is my favourite scripting language, but is dangerous at scale. I don’t love code that operates differently depending on tabs and/or whitespace, has implicit error handling, and does not require me to declare types. Too dangerous.

Pretty languages include:

SwiftUI - for macOS and iOS apps only. But I like it. Easy to read and pretty.
Go (of course) - a very simple language with limited magic. The go developers also seem to resist adding features that encourage magic or that is difficult to read and understand, such as complex inheritance or similar features that change how the standard library works.

Go has a simple but surprisingly broad standard library

Go has a relatively small and simple standard library. The tour of go and go by example will get you a very long way in understanding how go works.

It may be a quirk of history, and because of the networking focus (golang has always very much been a language driven by Google), but you can get a very long way just using the standard libraries.

Newcomers to go are often surprised that many go programmers recommend not using a web framework, for example. Instead, it’s usually best to use standard libraries, and only limited third party libraries where absolutely required. For example, I commonly use thin wrappers over the standard library such as the gorilla framework and sqlx, but mainly just stick to the standard library.

Go has fantastic built in options for error handling and concurrency

goroutines, golang channels, contexts, and the sync standard library (particularly sync/WaitGroup are incredibly powerful and useful tools for concurrent programming.

Go is agressively backwards and forwards compatiabile

I still use go code that I wrote years and years ago. It is one of the few programming language that I have ever used that allows me to recompile really old code without any changes. Try doing that with node shudder. I also like that most third party go libraries also follow this convention.

One caveat is that go has recently introduced go modules which has added some complexity when we update multiple libraries and modules as part of a larger project. However, go modules have brought us simpler and safer version control and semver compliance which is ultimately a good thing. In my experience, if you build with go using a third party go library, and update to the latest version of that third party library years later, you can be almost certain that your old code will still run.

Go has opinionated tools, meaning that most go code is simple and easy to read

If you use VSCode to develop go. And I my view you should. Every time you save a go file, VSCode updates the code to “standardised” spacing. This is a bit of a shock the first time it happens, but makes for very readable code across projects. I like it.

Go also does not have features that would encourage magic or would otherwise be difficult to read and understand. I also like that go runs very fast, but prioritises code readability over raw speed. The go developers even resisted introducing generics for many years. Generics are great, but I haven’t needed to use them. I built one library pre-generics which probably could be simplified by using generics, but I do not use generics in any of my existing code.

Testing is a first class feature of go

Any file ending in “_test.go” can contain test functions that are run with the go test command. This allows for, and pushes developers towards, simple test driven development.

Creating these tests files also exposes if a go file is too long and/or if our project needs refactoring. If all of the functions in a .go file cannot be relatively easily tested using its _test.go file, then we probably need to refactor.

Go has strict code constraints

I like static typing. I think that it makes code much easier to read, and has saved me from bugs more times than I can count.

If you create a go variable (or import a library) and don’t use it, then the go compiler won’t allow you to build the code until you delete the variable or library or update the code to use the variable or library. It sounds annoying but encourages clean coding.

Go has fantastic standard documentation

The golang standard library documentation includes examples for most libraries making them very easy to understand and use.

This is great, particularly because documentation outside the standard library is … not so great.

Go doesn’t let you take shortcuts

For example, to add a new item to a slice, we use the following go code:

s = append(s, newitem)

Why so verbose? Because adding a new item to a slice uses up computing resources in a relatively expensive way. Go refuses to hide this complexity.

Go requires explicit error handing

A common complaint about go is that it is littered with if err != nil lines. For example:

import "fmt"

func main() {
	for i := 0; i < 100; i++ {
		err := anythingButFourtyTwo(i)
		if err != nil {
			panic(err)
		}
	}
}

func anythingButFourtyTwo(i int) error {
	if i == 42 {
		return fmt.Errorf("so long and thanks for all the fish")
	}
	return nil
}

I love this because we need to explicitly handle every error. When you review a new go code base, this makes it much easier to understand how the code works, and critically what happens when there are errors.

Go is strange in the best possible way

Coming from a c and python background, I found the following both strange and awesome about go:

Goroutines, and sharing memory by communicating instead of communicating by sharing memory.
Use of interfaces generally, including in the standard library, such as the use of io.Reader and io.Writer.
Version control being effectively built in. The go ecosystem’s reliance on github is something to watch long term, but version control is relatively easy and go libraries generally follow the approach of the go standard library by rigorously complying with semver.
Layering contexts and error handling throughout large projects.
etc.

Some issues with go

Go is easy to learn but difficult to master

Golang has simple and incredibly powerful built in options for concurrency such as goroutines, golang channels, and the sync standard library (particularly sync/WaitGroup. However, it is very easy to create buggy code that:

creates thousands or millions of goroutines that use all of the machine’s memory;
doesn’t handle errors;
deadlocks;
attempts to close open channels; or
panics when it encounters an unexpected nil variable.

There are best practices coding approaches that we can use to avoid these problems, but learning these best practices only comes with experience.

Dark corners

Like any language, go has some dark corners that are best avoided if possible, including:

overuse or unnecessary use of interfaces, reflect, and the unsafe package;
building code that may use nil pointers (causing panics);
lazy goroutine creation and use, leading to memory leaks and attempting to close open channels;
memory leak risks when passing pointers between functions; and
lack of control over garbage collection.

During my go journey, so far I have struggled with (in this order, and particularly with relatively large codebases):

creating concurrent code that exits and restarts gracefully, and use of contexts;
finding and fixing memory leaks; and
garbage collection and opaque memory management.

Golang tooling is very good, but sometimes its garbage collection seems a little opaque. It is relatively easy to write go code that uses a lot of memory (or all the memory) and I only learned over time to use pprof effectively to find memory leaks, and how to code to use a very small memory footprint.

Once coded well, go code is incredibly memory efficient. It is usually quick and easy to reach the point where the CPU, or filesystem or network i/o, become the bottlenecks instead of available memory (i.e. the speed of the running go code is no longer the performance choke point).

Limited great books and documentation outside of the standard library

The standard library documentation is fantastic. There are also a lot of go books and articles. However, because golang is not as widely used as python (for example) once we move away from simple uses of the standard library less great documentation exists.

For example, I found that learning to use go’s simple and powerful concurrency features to build complex apps, with multiple error fallbacks, was more challenging than expected. A really great go concurrency at scale cookbook would have saved me a lot of time.

Go articles also often focus on less complex apps that can ingest all data into memory. For example, many golang articles and third party golang libraries read the entire contents of a file into memory by default rather than streaming and then processing individual bytes and/or lines of code in a file. This in spite of the fact that the go standard library provides robust tools and processes to allow for streaming such information (such as io.Reader) as bytes or lines.

My frustration is that go does allow us to easily build complex apps that ingest huge amounts of memory. However, the go books and articles that I have read do not show us the best practices for how to do so. All of the issues discussed above can be managed with best practice coding, but I have not read any books that describe how to write really robust go apps that avoid these issues.

I have learned how to avoid these issues over time by improving my go coding by trial and error. However, it has always been clear to me that any issues that I have with go relate to my lack of understanding of how to code well rather than any underlying problems with the language itself.

macOS golang backend and native SwiftUI frontend using http long polling

Thu, 28 Mar 2024 00:00:00 +0000

calmdocs/SwiftPollManager

Run a golang binary embedded in a native macOS SwiftUI app.
The golang binary and SwiftUI app communicate via http long polling.
I hope that you find it useful.

1. Websockets on macOS is flakey

A while ago, I released calmdocs/SwiftStreamManager, which does the same thing, but communicates via websockets.

While calmdocs/SwiftStreamManager works as advertised, if used on a macOS laptop that is often woken and put to sleep, after a long period of time the Swift websocket connection may fail, requiring an automatic restart of the golang binary. This is an issue with the underlying macOS Swift websockets library. Please consider using calmdocs/SwiftPollManager instead which communicates via http long polling.

2. Messages between the golang binary and SwiftUI app should be encrypted …

Almost all of the feedback that I recived in relation to calmdocs/SwiftStreamManager was in relation to whether encrypting the messages sent between the golang binary and SwiftUI app is necessary. I think that it is. macOS does not allow you to use https (i.e. encrypted) connections without significant complexity. However, connecting the SwiftUI and golang apps via http is relatively simple.

calmdocs/SwiftPollManager allows the creation of a Diffie–Hellman Key Exchange (DHKE) connection between the SwiftUI app and golang app using the calmdocs/SwiftKeyExchange swift library and calmdocs/keyexchange go library. The SwiftUI app sends its public key as an argumant to the golang app, and the golang app then sends its public key to stdOut as a PEM message for the SwiftUI app to read.

3. Encryption is optional

calmdocs/SwiftPollManager also contains simple implementation instuctions If you want to use this library without encypting or if you want to use your own encryption.

Run a golang binary embedded in a native macOS SwiftUI app

Tue, 24 Oct 2023 00:00:00 +0000

calmdocs/SwiftStreamManager

Run a golang binary embedded in a native macOS SwiftUI app.
The golang binary and SwiftUI app communicate via encrypted websocket messages.

1. Setup

Create a new macOS Swift Xcode project:

File -> Add Packages … -> https://github.com/calmdocs/SwiftStreamManager
Select the checkbox at Target -> Signing & Capabilities -> App Sandbox -> Network -> Incoming connections (Server).
Select the checkbox at Target -> Signing & Capabilities -> App Sandbox -> Network -> Incoming connections (Client).

2. Example

(a) Create our go binaries (for amd64 and arm64)

Run the following commands:

git clone https://github.com/calmdocs/SwiftStreamManager
cd SwiftStreamManager/pkg/gobinary
GOOS=darwin GOARCH=amd64 go build -o gobinary-darwin-amd64 && GOOS=darwin GOARCH=arm64 go build -o gobinary-darwin-arm64

Drag the gobinary-darwin-amd64 and gobinary-darwin-arm64 files that we just built into our new macOS Swift Xcode project.

(b) Replace ContentView.swift

In our new Swift Xcode project, replace ContentView.swift with the following code:

import SwiftUI

import SwiftStreamManager
import SwiftWebSocketManager
import SwiftProcessManager

public struct CustomStorageItem: Codable, Identifiable {
    public var id: Int64
    
    let error: String?
    let name: String
    let status: String
    let progress: Double

    enum CodingKeys: String, CodingKey {
        case id = "ID"
        case error = "Error"
        case name = "Name"
        case status = "Status"
        case progress = "Progress"
    }
}

struct ContentView: View {
    @ObservedObject var itemsStore: ItemsStore = ItemsStore()
     
    var body: some View {
        List {
            HStack {
                Button(action: {
                    itemsStore.sm.publish(
                        itemsStore.sendStream!,
                        type: "addItem",
                        id: "",
                        data: ""
                    )
                }, label: {
                    Image(systemName: "plus")
                })
                Spacer()
            }
            ForEach(itemsStore.items) { item in
                HStack{
                    Text("\(item.name) (\(item.status))")
                    Spacer()
                    Button(action: {
                        itemsStore.sm.publish(
                            itemsStore.sendStream!,
                            type: "deleteItem",
                            id: String(item.id),
                            data: ""
                        )
                    }, label: {
                        Image(systemName: "trash")
                    })
                }
            }
        }
    }
}

class ItemsStore: ObservableObject {
    private let binName = SystemArchitecture() == "arm64" ? "gobinary-darwin-arm64" : "gobinary-darwin-amd64"

    @Published var items: [CustomStorageItem] = [CustomStorageItem]()
    
    // StreamManager
    @Published var sm: StreamManager
    @Published var sendStream: WebSocketStream?
  
    init() {
      
        // Initialise StreamManager
        self.sm = StreamManager(
            KeyExchange_Curve25519_SHA256_HKDF_AESGCM,
            baseURL: URL(string: "ws://127.0.0.1")!,
            port: Int.random(in: 8001..<9000)
        )
        self.sm.addPIDAsArgument("pid")
        self.sm.addBearerTokenAsArgument("token")
        self.sm.addPortAsArgument("port")
        
        // Start binary and subscribe to a websocket stream
        self.sm.subscribeWithBinary(
            streamPath: "/ws/0",
            binName: self.binName,
            withPEMWatcher: true,
            standardOutput: { result in
                print(result)
            },
            messages: { message in

                // Decrypt and decode
                guard let newItems: [CustomStorageItem] = try? self.sm.decryptAndDecodeJSON(
                    message: message,
                    auth: self.sm.authTimestamp  // Use the current time since 1970 in milliseconds as the default key exchange auth key.
                ) else {
                    return
                }

                // Update items
                DispatchQueue.main.async {
                    self.items.replaceInPlace(items: newItems)
                }
            },
            onConnected: {
                
                // Create a second (sending) stream subscribed to a different path
                DispatchQueue.main.async {
                    self.sendStream = try! self.sm.stream(streamPath: "/ws/1")
                    self.sm.subscribe(self.sendStream!,
                        //messages: { message in
                        //    print(message)
                        //},
                        errors: { err in
                            print(err)
                        }
                    )
                }
            }
        )
    }
}

3. Security

We have been as conservative as possible when creating this library. See the security details available on the calmdocs/SwiftKeyExchange package page. Please note that you use this library and the code in this repo at your own risk, and we accept no liability in relation to its use.

Can banks use cloud services?

Tue, 14 Jul 2015 00:00:00 +0000

The Australian banking regulator (APRA) just released an information paper on ‘shared computing services’, including cloud computing. Can banks still use cloud services?

1. The banking regulator (APRA) has just released an information paper that deals with this question

Can Australian banks (and other APRA regulated Authorised Deposit-taking Institutions (ADIs))[1] use cloud services?

APRA has just released a new information paper (Outsourcing Involving Shared Computing Services (including Cloud)) regarding what is required of banks.

The use and processing of data by banks is actually very complicated once we get into the weeds. Particularly when we are dealing with global banks that must satisfy all of the applicable regulators in every country where we operate.

And while the information paper is not all that long, it provides quite detailed advice. Even the footnotes have interesting implications for banks. I am just scratching the surface here.

2. Banks must impose protections on suppliers as soon as data leaves bank-run data centres

That being said, APRA’s new information paper (Outsourcing Involving Shared Computing Services (including Cloud)) succinctly clarifies that it is important that banks implement appropriate measures to ensure that data hosted and/or processed outside of the bank’s internally controlled data centres must be appropriately protected.

The information paper clarifies APRA’s view of when banks must impose the full weight of protections (both technical and legal) in relation to external hosting arrangements. For example, APRA previously stated that banks must closely monitor the implementation and use of any ‘cloud’ arrangements. But there seems to have been some confusion, because there is no globally accepted meaning for the word ‘cloud’.

3. APRA is now avoiding imposing ‘cloud’ rules because ‘cloud’ is a marketing term not a technical term

In the information paper, APRA refers to bank obligations in relation to ‘shared computing services’ rather than ‘cloud’ services. As APRA says in the information paper:

The term ‘cloud computing’ is used to describe a broad variety of arrangements.

Some people may think that ‘cloud’ arrangements include all external hosting arrangements. Others may think that ‘cloud’ arrangements are arrangements where customers sign up with suppliers that have inadequate legal protections, and/or reserve the right to send (or process) customer data anywhere in the world.

APRA deliberately avoids this problem by stating that it is important that all ‘shared computing services’ be appropriately protected. APRA explicitly states that it is important that banks impose appropriate protections in relation to all external hosting services, including where entire data centres are shared with third parties. APRA is advising banks to protect this data, and implement appropriate security protections, no matter how or where this data is stored and processed. Even if each bank merely operates their own computer hardware within data centres that are only shared with other financial institutions (that are also regulated by APRA).

4. More critical bank data requires more robust bank and supplier protections

The information paper sensibly states that high risk arrangements require more protections than low risk arrangements. For example, APRA states that ‘un-trusted’ environments (where an APRA-regulated institution is unable to enforce its IT security policy) may have ‘heightened inherent risk’.

At the extreme end, APRA refers to ‘systems of record’ that hold information ’essential to determining obligations to customers (such as customer identity, current balance/benefits and transaction history)’.

But APRA worries that:

In light of weaknesses in arrangements observed by APRA, it is not readily evident that risk management and mitigation techniques for public cloud arrangements have reached a level of maturity commensurate with usages having an extreme impact if disrupted. Extreme impacts can be financial and/or reputational, potentially threatening the ongoing ability of the APRA-regulated entity to meet its obligations.

And APRA then states that:

APRA’s stance aligns with the position of other international financial regulators who also question the appropriateness of transitioning systems of record to a public cloud environment.

5. Has APRA clarified or changed the rules?

APRA seems to be of the view that this information paper is a lot like the last round of clarifications to the privacy laws in Australia. Organisations that construed their privacy law obligations as narrowly as possible needed to scramble to ‘uplift’ their privacy compliance regimes when the privacy laws were ‘clarified’. But organisations that took into account the original intent of the privacy laws, and built their processes conservatively, did not need to change their processes much at all.

This is a similar situation. APRA seems to be of the view that APRA hasn’t imposed new obligations on the banks. But has instead clarified the scope of existing bank obligations (in line with other regulators such as the Singapore banking regulator, that has recently clarified some of their rules in relation to the use of ‘cloud’ services by Singapore banks).

6. What happens now?

In the information paper, APRA goes into great detail regarding how some financial institutions may have (in APRA’s view), failed to implement appropriate external hosting protections. For example, APRA lists the following ‘observed weaknesses’ in existing bank processes:

high-level risk descriptions that lack clarity or are documented as statements of control weaknesses;

lack of consideration of critical and/or sensitive IT assets which are accessible from the shared computing service;
inadequate consideration of the sensitivity of data (collectively and at the individual field level) when considering implementation solution options for shared computing services;
cursory risk assessments which fail to consider specific risks and any changes to the risk profile; and
limited due diligence and assurance activities undertaken, with heavy reliance placed on provider attestations and/or usage by other organisations.

There will be much parsing of this information paper over the next few months by the banks. And APRA will almost certainly closely scrutinise what constitutes low risk arrangements, and what constitutes high risk arrangements, to confirm that banks have adequate protections in place for both types of arrangements.

APRA may also scrutinise how robust banks’ internal processes, and supplier agreements, are when banks use ‘shared computing services’ to store and process different types of bank data. Particularly given that APRA states in the information paper that:

APRA’s review of these arrangements has identified some areas of weakness, reflecting risk management and mitigation techniques that are yet to fully mature in this area.

Footnotes:

This post originally appeared at iainmclaren.com. These opinions are mine. They are not necessarily those of my employer. And this is not legal advice. Please seek it if you need it.

Suppliers make more money by signing tougher contracts

Wed, 18 Feb 2015 00:00:00 +0000

Immature suppliers smash their standard contract terms and conditions through the bureaucracy of their enterprise customers. But mature suppliers sell more services (and make more money) by signing “real” contracts, and by taking real responsibility for providing great services.

1. Enterprise customers will buy (and pay more for) services backed up by “real” contracts

Mark Cranney from a16z recently wrote If SaaS Products Sell Themselves, Why Do We Need Sales?. Mark argues that:

Some people think the sales force’s job is to communicate value to customers. To these people, sales is about buying a bunch of search ad words or mouthpiecing a company’s message.

They’re wrong.

I couldn’t agree more (other than “mouthpiecing” - I don’t think that’s a word - but anyway). Your supplier sales consultant will tell you that in the new Agile world customers should sign whatever ludicrous terms and conditions are placed in front of them.

That is a fallacy. Large enterprise customers will continue to favour service providers that are willing to sign contracts that provide the customer with robust protections (i.e. are worth the paper that they are written on). As Mark says:

… [E]nterprise/SaaS sales requires a well-developed process and guidelines … [and] the principles remain the same even though the domains have changed. And whether SaaS entrepreneurs like to admit it or not, the new enterprise customer is a lot like the old enterprise customer.

2. And supplier sales executives can close more deals by challenging their business teams to sign “real” contracts

In order to sell services to enterprises, Mark argues that there are three things that every enterprise customer wants to know:

Why should I do anything?
Why your solution (vs. your competitors’)?
Why now (vs. investing in another area of our business)?

I think that is true, but I have found that a large number of enterprise customers already have satisfying answers to these questions. In my experience, the problem that customers face is that many suppliers now seriously argue that suppliers should take little or no legal responsibility for the services that they provide to enterprise customers.

However, large enterprise customers actually expect that the supplier’s services will be provided under terms and conditions that require those suppliers to accept responsibility, in a contract, if their service does not work.

3. Therefore, suppliers that are willing to pay for failure can make more enterprise sales and charge higher margins

Global suppliers make higher margins as they move up the value chain from low risk time and materials services to higher risk fixed price outcome based contracts.

We saw this play out with global technology suppliers during the last 15 years. These suppliers initially made people in lower-wage countries available to customers on a time and material basis. But as more and more players entered this market, the margin that these suppliers charged reduced.

These global technology suppliers soon realised that the only way to maintain high margins was to take more responsibility for customer business outcomes (rather than just providing personnel, charging a daily rate, but taking no responsibility for customer business outcomes). They moved up the value chain.

The same thing is about to happen with externally hosted (i.e. cloud) services. Cloud service providers have moved up the value chain by moving from IaaS to PaaS to SaaS.

But cloud providers, for example, are now differentiating themselves along another axis. Enterprise customers will pay more for cloud providers that are willing to take responsibility, and move up the value chain, by signing terms and conditions that require them to take real responsibility for service failures.

4. The supplier choice: accept risk or race to the bottom

Don’t get me wrong. Enterprise customers are very cost sensitive, but enterprise customers will usually favour suppliers that provide real contracts, with real service levels. Enterprise customers will favour suppliers that are willing to accept real responsibility if things go wrong.

And enterprise customers will pay for this privilege. Maybe suppliers that sign real contracts won’t be swimming in gold coins like Scrooge McDuck. But they won’t be locked into a race to the bottom on price with their competitors. And they won’t leave millions of dollars on the table to be snapped up by their competitors.

This post originally appeared at iainmclaren.com. These opinions are mine. They are not necessarily those of my employer.

No middle ground - why CIOs thrive or die

Tue, 10 Feb 2015 00:00:00 +0000

Customer business teams are outsourcing their technology to cloud providers while avoiding or ignoring their internal technology teams. Great Chief Information Officers (CIOs) who embrace wider business responsibilities will thrive. The other CIOs won’t survive.

1. A CIO can no longer be a pure technologist

Last year, CIO Magazine published an article titled “How to Be the CIO of Tomorrow”:

What does the future CIO look like? He or she probably won’t have a technology background. However, this new kind of CIO will not only have to lead a technical staff but will also be a leader of leaders in the boardroom.

The problem with the argument that CIOs don’t need a technology background is that while CIOs can no longer be mere technologists, they still need a deep understanding of technology to be effective. Particularly as technology is now so integral to all businesses, and because there is no longer any real distinction between customer business team responsibilities and customer technology team responsibilities. For example, this is right:

Viewing the business through the eyes of the customer is not something that we tend to do as CIOs. We tend to spend a lot of time talking to supporting functions and internal operations. But are we prepared to take an outside-in view of the business? The typical CIO plays within a certain set of boundaries, or a box. Are we willing to step outside of that and drive cross-functional initiatives? If we stick within an IT boundary, we’re going to miss the digital transformation agenda.

2. But many CIOs have lost control of their technology and information systems

The argument in the article is that the role of the CIO has not been diminished, but the following quote is telling:

I see the CIO as taking the guardian or custodian role, so we don’t have shadow IT organizations popping up everywhere. The days of IT strategy getting decided just by the CIO, and the CIO having to convince the CFO or CEO, are gone. The CIO has to be able to take input from all sources and manage and direct the conversation at board level to ensure that we’re getting a 360-degree view of what we should be doing.

In other words, many CIOs have lost control over how technology (and information systems) are used in their organisations. Business owners are moving technology and business functions to the cloud while avoiding, or outright ignoring, their own technology teams.

3. Because increasingly our technology systems are black boxes that we don’t understand

Company technology systems will become brittle and insecure if no-one in the company has a deep understanding of technology because company employees will inevitably just believe supplier sales spin. The CIO Magazine article argued that this reliance on suppliers is a good thing:

Technical acumen, which has always been seen as a core field for CIOs, I believe will be increasingly acquired from different sources, such as partners, vendors, customers. Some aspect of IT strategy will be crowd-sourced from the business

But it is actually a major problem.

4. And a CIO who doesn’t understand technology is as dangerous as a CFO who doesn’t understand finance

In other words, if a supplier came to you and offered to run all of your finance functions, would you fire your Chief Financial Officer? Would you accept the risk that you no longer have anyone in your organisation who understands your business finances?

I think not.

And given that technology is now a critical part of all businesses, if your CIO does not understanding technology in general, and your technology in particular, then that is as risky as your CFO not understanding finance.

5. Great CIOs have a deep understanding of their technology stack, their industry, and their business

CIOs now need to know more to thrive or simply survive. And a great CIO needs to have a strong technology background. Because how can you advise on technology if you don’t understand it?

A great CIO also needs to have a deep understanding of the business and the industry in which the CIO works. A great CIO builds internal and external relationships, understands the business’ internal and external systems at a granular level, and works with internal and external stakeholders to constantly improve business processes. And a great CIO quite simply eats the cloud for breakfast.

Mmmmn tasty.

This post originally appeared at iainmclaren.com. These opinions are mine. They are not necessarily those of my employer.

Zen and the art of drafting simple contracts

Wed, 04 Feb 2015 00:00:00 +0000

Simple contracts are quick and cheap to draft and negotiate. But if our contract is too simple then we risk project failure. How do we draft the simplest possible contract?

1. Time is actually money (in this context at least)

Everything should be made as simple as possible, but not simpler

– Albert Einstein (attrib.)

The feedback that I received in relation to my article on “Agile” contracts was fantastic. And when I say fantastic, I mean half the comments were very positive, and the other half of the respondents thought that I had taken leave of my senses. Because the problem with “Agile” is that it is often used as an excuse not to project manage at all, or draft any form of contract at all.

So how do we draft the simplest possible contract that is also fit for purpose?

2. The larger and risker the project, the more project management and detailed contract structure we require

We need to project manage, including by agreeing up front what the customer is buying. But our contracts should be as short as possible, because the longer the contract, the longer it will take to negotiate. And in spite of what the poets say, time is in fact money - in this case at least.

In other words, simple contracts are relatively quick and cheap to draft and negotiate, but not all contracts should be short and simple. Because if our contract is too simple, then we risk project failure.

3. Use time and materials contracts for simple projects, and fixed price contracts for everything else

There are two contracting models:

Time and materials contracts: where the supplier provides their people at a fixed daily rate (or hourly or monthly rate), but the supplier does not guarantee in the contract that their people will produce any particular outcome. We use time and materials contracts for projects that are low value and not complex (or risky).
Fixed price contracts: where the supplier only gets paid if the supplier achieves specific outcomes. We use fixed price contracts for projects where it is worth the time to develop detailed specifications.

4. Don’t over-engineer or under-engineer the contract

The top left and bottom right quadrants in our table above show what happens if we “over-engineer” our contracts by using time and materials contracts for large complex projects, or “under-engineer” our contracts by using fixed price contracts for projects that are low value and not complex (or risky).

We don’t:

Over-engineer by using fixed price contracts for projects that are low value and not complex (or risky), as the time invested in contract negotiations is not worth the return. For example, it makes no sense to pay more to negotiate the contract than its value (e.g. if we invest $1m to draft and negotiate a contract with a value of $10,000), unless what we are buying is critical to the business, or if the supplier providing inadequate services might cause damage to the business.
Under-engineer by using time and materials contracts for large critical projects, as if we do not agree up-front what the supplier will provide to the customer, the customer risks massive cost blow outs, as the supplier will take no risk if there is a disconnect between what the customer thought that they were buying and what the supplier actually provides. The supplier will take no risk, and if the customer is not happy with the outcome of the project, the supplier will just charge more to complete the project.

5. Fixed price contract tools: modularising, catalogues, and Agile contracts

It takes a significant amount of time to negotiate a large, fixed price contract. Therefore, once we have chosen a fixed price contract, there are three tools that we can use to simplify the contract drafting process, and to make implementing the contract as simple as possible:

“Modularised” contracts: are contracts that are split into smaller “chunks” or milestones with specific deliverables, customer acceptance tests, and payment gateways once each deliverable is delivered to the customer and accepted by the customer. As discussed in my “Agile” contracts article, some phases of the contract (separated by milestones) may be time and materials phases, and others may be fixed price.
Catalogues: contemplate the customer purchasing a (relatively) large number of “widgets” that are well defined in the contract. For example, if the customer signs a contract to purchase a large number of laptops on an ad-hoc basis, it is worth spending the time to negotiate a contract that specifies exactly what the customer will be getting (e.g. laptop specifications, service levels, delivery times, etc.). Catalogues can work for any type of well defined service or deliverable that the customer may purchase in bulk over time. Catalogues can even work for business process outsourcing services and similar deliverables.
“Agile” contracts: are modular contracts with fixed deliverables and phases, that allow the customer and supplier to use an Agile process for some or all of the contract phases, but still requires the supplier to produce “fixed” deliverables. The key is to ensure that our business team does noes not ask for an “Agile” contract with no milestones or deliverables specified in the contract. That’s just another way of proposing using a time and materials contract for a large critical project.

In summary, we can still keep our contracts simple. But it is a mistake to use simple time and materials contracts for large (or otherwise critical) projects.

This post originally appeared at iainmclaren.com. These opinions are mine. They are not necessarily those of my employer. Thanks to Alex Nathan for working with me to create the above matrix, and for reviewing drafts of this post.

Can you draft an "Agile" contract?

Wed, 28 Jan 2015 00:00:00 +0000

One of the four core Agile values is “customer collaboration over contract negotiation”. So is it possible to negotiate an “Agile contract” or is the whole concept nonsensical?

1. What is “Agile” and is the whole concept of an “Agile contract” nonsensical?

There are four core Agile values:

Individuals and Interactions over Processes and Tools
Working Software over Comprehensive Documentation
Customer Collaboration over Contract Negotiation, and
Responding to Change over Following a Plan

Dave Thomas, one of the original creators of Agile, laments in an article that the term has been subverted by consultants and suppliers to the point where it has become meaningless.

I agree. And yes I am a consultant. And yes I see the irony here. Given that one of the core Agile values is “customer collaboration over contract negotiation”, there is a strong argument that the whole concept of “Agile contracts” is ridiculous. As Dave says in his article:

I haven’t participated in any Agile events, I haven’t affiliated with the Agile Alliance, and I haven’t done any “agile” consultancy. I didn’t attend the 10th anniversary celebrations.

Why? Because I didn’t think that any of these things were in the spirit of the manifesto we produced. Having conferences about agility is not too far removed from having conferences about ballet dancing, and forming an industry group around the four values always struck me as creating a trade union for people who breathe.

And, unfortunately, I think time has proven me right. The word “agile” has been subverted to the point where it is effectively meaningless, and what passes for an agile community seems to be largely an arena for consultants and vendors to hawk services and products.

But we still need contracts with our suppliers. We still need to lock down the fees, and specify in writing who owns the intellectual property rights in what the supplier produces. So what type of contract should we build if we want the process to be Agile?

2. Agile’s power is in its flexibility. But Agile projects do have an underlying structure.

There are twelve principles of Agile software:

Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.
Welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage.
Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.
Business people and developers must work together daily throughout the project.
Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.
The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.
Working software is the primary measure of progress.
Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.
Continuous attention to technical excellence and good design enhances agility.
Simplicity–the art of maximizing the amount of work not done–is essential.
The best architectures, requirements, and designs emerge from self-organizing teams.
At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behaviour accordingly.

If a customer builds an in house team of developers, then we can follow the above process, and avoid contracts completely. But we have two choices if we need to integrate supplier engineers into our team or project:

a time and materials contracting model; or
a fixed price contracting model.

The core issue is actually whether the customer and the supplier need to agree the specifications for the project in writing up-front, and (if so) how detailed these specifications need to be.

3. Our two choices - time and materials contracts or fixed price contracts

It sometimes seems more complicated than this. But, actually, if customers utilise people (e.g. software developers) who are employees or contractors of the suppliers, then they do so under one of the following two contracting models:

time and materials contracts, where the supplier provides their people at a fixed daily rate (or hourly or monthly rate), but the supplier does not guarantee in the contract that their people will produce any particular outcome; and
fixed price contracts, where the supplier only gets paid if (hopefully when) the supplier has achieved specific outcomes.

There are only two models. With fixed price projects, the specifications are agreed in writing up front, and with time and materials projects, they are not.

All contracts for the supply of people are a variation, and sometimes a combination, of the above two contracting models. For example, under the traditional waterfall contracting model, a customer might pay a supplier (all under one contract):

a daily rate for some initial “general consulting” work;
then a fixed price for the supplier to produce a detailed design document for the project; and
then a fixed price (specified in that design document) to implement the project (e.g. develop software) or to provide other output or outcome specified in the applicable contract documents. The supplier is traditionally paid milestone payments (specified in the design document) after each part of the project is completed by the supplier and accepted by the customer.

4. So does the Agile “customer collaboration over contract negotiation” requirement mean that we can only use time and materials contracts?

I don’t think so. A better question is how detailed do the specifications in our contract documents need to be. An Agile approach calls for these specifications to be as high level as possible to allow the teams to run the project in as flexible a manner as possible.

For example, time and materials contracts are appropriate for really small projects, such as the initial consulting part of the project described above. But the problem with time and materials contracts is that the supplier does not commit to deliver any specified outcome within any specified period of time.

Therefore, signing a time and materials contract limits the amount of time that we need to negotiate specifications (and contracts) up front, but significantly increases the risk that the price, timeline for delivery, and outcome will be nothing like what the customer originally expects.

In other words, if we ask a supplier to implement a multi-million dollar project under a time and materials contract, then we court failure. And a customer would be negligent if it risked running a multi-million or billion dollar toll road or other major infrastructure project as a pure time and materials project.

5. But isn’t fixed price the opposite of Agile?

Again, no, I don’t think so. Let’s look at the fundamental differences between time and materials and fixed price contracts.

For time and materials contracts, at the start of the project, the customer and supplier may estimate the time, and therefore cost, of producing a particular outcome. But if we sign a time and materials contract without taking the time to draft clear specifications (and agree on project timelines, costs, and deadlines), the customer takes all of the risk that this estimate is wrong. Or that the deliverables will be nothing like what the customer originally expected. The customer will then need to pay for the supplier’s additional time, and the supplier is not penalised if the project takes longer than the supplier originally estimated. In fact, if the project is delayed, the supplier makes more money even if this delay is caused by the supplier’s delays or mistakes.

On the other hand, a fixed price contract (for example that requires the supplier to follow a traditional waterfall model) requires the customer and supplier to spend a lot of time up-front developing specifications. This is often the opposite of an Agile process. The negotiation process can be clunky, and any change to the specifications can be a tedious process that ignores the value of principle 11:

The best architectures, requirements, and designs emerge from self-organizing teams.

But the advantage of a fixed price project is that it requires the supplier to commit to a price and timeline.

6. A fixed price Agile working example

Let’s go back to our example project. The customer needs both flexibility and for the supplier to have “skin in the game” by committing to specific deliverables and timelines. In other words, we need to agree how Agile principle 12 will work for each project:

At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behaviour accordingly.

So for our example project we can implement an Agile process as follows:

The customer pays the supplier a daily rate for some initial “general consulting” work. Pure Agile. Simple.
Then the customer pays the supplier a fixed price for the supplier to produce a detailed design document. We still have no detailed specifications so we can follow our Agile process. But the customer and supplier can agree a fee cap and timeline for providing this design document. If we follow rule 4: “Business people and developers must work together daily throughout the project”, then any scope (and pricing) change process should be quick and simple.
Then the customer can pay the supplier a fixed price (specified in that design document) to produce the final deliverables specified in the applicable contract documents. The customer can also pay the supplier milestone payments (specified in the design document) when each part of the project is delivered and accepted by the customer. Again, if the customer and business teams follow an Agile process, then changes to the pricing and milestones should be quick and simple.

7. But that’s not a “true” Agile process!

Yes, OK, you’re probably right. But if by “true” Agile, you mean that all projects should be time and materials projects with no specifications agreed up front in writing, and no supplier delivery risk, then there is no chance that sensible customers will agree to this approach for large projects that involve external suppliers. It is not realistic to expect customers to commit to pay suppliers tens of millions of dollars with no specifications agreed upfront, no guarantee that what the customer receives will be anything like what the customer originally wanted, and no guarantee that the price and timeline will be anything like the original supplier “estimate”.

But Agile does not mean no project management or controls. Quite the opposite. An Agile contract is a contract that provides just enough structure (and specifications) to ensure that both the supplier and customer act as the other expects. But an Agile contract should then get out of the way of the individuals who actually deliver what the customer wants. That is the power of Agile principle five:

Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.

This post originally appeared at iainmclaren.com. These opinions are mine. They are not necessarily those of my employer. Thanks to Josh Morris and Alex Nathan for reviewing drafts of this post.

Why large companies don't invest in security

Mon, 19 Jan 2015 00:00:00 +0000

Data breaches are currently embarrassing but not crippling. But major data breaches will destroy companies once instant anonymous digital transactions become mainstream.

1. Even the largest company data breaches to date have been relatively minor. No really.

The scale of the data breaches listed in the following infographic is shocking. Hundreds of millions of users have been harmed by major data breaches over the last 10 years.

However, the large organisations listed in this infographic have shrugged off these breaches and are still running. A few senior executives have been pushed out, or there have been some financial losses, but these organisations have been inconvenienced but not crippled. And third parties, such as customers whose credit card details have been exposed to the world, have been (often seriously) inconvenienced, but not financially crippled or bankrupted in large numbers.

This is going to change. And soon.

2. Companies have not prioritised security because the consequences of major security breaches have usually been relatively minor

Sony was hacked in the nastiest possible way in late 2014. But as Ben Thompson (article behind a paid firewall) says:

There is a serious problem when it comes to Internet security: companies and their managers are simply not incentivized to prioritize security. Putting in place real security is difficult, it’s expensive, and it’s like proving a negative when it comes to ascribing value: what is it worth to have not yet been hacked?

Sony Pictures Entertainment executive director of information security Jason Spaltro actually says so himself in this article on CIO.com:

Noncompliance is a fact of life as the list of security and privacy regulations grows. The key is knowing how to comply just enough so that you don’t waste your time or bankrupt your company.

Ben Thompson called this CIO.com article “surreal” (article behind a paid firewall), but this attitude is actually common practice in large organisations because paying for security, like paying for anything in organisations, requires that security to pass a cost/benefit analysis. Organisations have (correctly) assumed to date that the worst outcome of implementing inadequate, or frankly negligent, security is that you might lose a bit of money and some of your senior executives might go. Therefore, why would those organisations spend time and money implementing quality security? It is very expensive and time consuming to implement security measures that do not hamper the business from doing business.

The answer? Future security breaches will actually destroy some companies once anonymous digital currencies become mainstream.

3. Mainstream anonymous digital transactions will make money laundering much easier

William Gibson is a science fiction writer best known for coining the term “cyberspace”. In 1982, Gibson published a short story called Burning Chrome. In that story, Gibson predicted that the rise of anonymous digital credit (such as bitcoin transactions) would change the nature and scale of crime, and result in data breaches that destroy companies. Burning Chrome is one such story:

“He was fading fast, and smart money was already whispering that the edge was off his game. He needed that one big score, and soon, because he didn’t know any other kind of life, and all his clocks were set for hustler’s time, calibrated in risk and adrenaline and that supernal dawn calm that comes when every move’s proved right and a sweet lump of someone else’s credit clicks into your own account.”

Gibson foresaw that the missing link, that will make widespread catastrophic security breaches possible, is easier money laundering.

4. And company data breaches will be catastrophic once money can be easily laundered

Let’s assume that it is only a matter of time before digital currencies such as bitcoin, that can be stored and transmitted anonymously, become mainstream. These currencies, or other forms of instantaneous anonymous transactions, will at least initially operate alongside traditional currencies backed by national governments.

But once effectively unlimited money can be stored and transmitted anonymously, the nature and scale of crime will change. As Gibson points out, now that massive data breaches are commonplace, the only thing stopping massive, company destroying fraud is that it’s not money that hackers steal. It’s credit card details.

Credit cards can be cancelled, and money transfers can be tracked. Currently, hackers generally sell credit card numbers in blocks to criminals who use those card numbers to make small transactions, like buying electronic goods, or store gift cards. Small time criminals buy relatively cheap goods that have a resale value, and then resell those goods for cash. Criminals never make real money that way.

But when a data breach is no longer a credit card breach, then hackers will target systems and companies that hold digital currency. Because in a world of instant payments, all of our money, and our customers’ money, can be instantly and irretrievably stolen. Taking us and our customers with us. Hackers will be able to access company systems because of existing lax company security standards. And hackers will be able to steal vast amounts of money as soon as anonymous digital currencies become mainstream.

We will soon see this play out in reality. And as implementing secure systems takes time, companies that have not invested in security may not survive.

This post originally appeared at iainmclaren.com. These opinions are mine. They are not necessarily those of my employer.

How to hire a superstar

Wed, 24 Sep 2014 00:00:00 +0000

Superstars are specialists

1. Superstars are smart and get things done

In 2000, Joel Spolsky wrote the Guerrilla Guide to Interviewing. By far the most sensible document that I have ever read about employing people. Some parts of Joel’s article only apply to the technology industry. But it is still worth reading the whole thing. As Joel says:

In principle, it’s simple. You’re looking for people who are:
- Smart; and  
- Get things done.

Candidates with zero out of two of these traits are, frankly, a waste of money. As are smart ineffective people. But the worst hires are those who are not smart, but still get things done. These people actively break things. They can break your business.

Testing for smartness is relatively easy. But testing whether (or not) your candidate can actually get hard things done involves determining whether (or not) that person is a specialist. A genuine specialist. Let me explain.

2. The simple smartness test

If your candidate survives the resume cull, and has made it into the room with you, then the smartness (not just intelligence) test is quite simple.

We sometimes think it’s hard. Or overcomplicate our decision making process. But as Joel says (seriously, read the whole article), if we are in any doubt about a candidate then we should simply not employ them:

You’re going to see three types of people in your interviews:
- At one end of the scale, there are the unwashed masses, lacking even the most basic skills for this job. They are easy to ferret out and eliminate, often just by asking two or three quick questions. 
- At the other extreme you’ve got your brilliant superstars ... 
- And in the middle, you have a large number of “maybes” who seem like they might just be able to contribute something.

Sorry for repeating myself, but I’ll say it again. If in doubt, don’t hire:

The trick is telling the difference between the superstars and the maybes, because the secret is that you don’t want to hire any of the maybes. Ever..

I have made this mistake and it’s a big one. When we need to hire. And one candidate is both fine on paper, and better than the other candidates. But that person does not have a passion for the business. Or your team is unimpressed. Or hiring them just doesn’t feel right. Then just don’t hire them.

3. Only specialists get hard things done

Our prospective new hire may be smart, but can he or she get things done? We look for examples of hard achievements or accomplishments. We look for victories.

This might be getting into (and completing) a competitive University course. Or military service. Or setting up and running a successful business or charity. Or being an accomplished concert violinist. And it might be multiple achievements or accomplishments. We look for anything genuinely competitive or hard.

And to win. Or more to the point, to have won. Our candidate must have learned specialist skills. Our superstar candidate will already be a specialist CEO, or a specialist engineer, or an expert management consultant, or an accomplished musician.

Only specialists get hard things done.

4. A superstar may be a specialist in another field

Or possibly a few fields. But avoid the candidate who is apparently generally (and generically) great. Avoid potential hires who are not, and have never been, specialists in any field.

And obviously we would prefer that our new hire has a demonstrated passion for their new job (or at least industry). But this is not an absolute requirement, particularly for relatively junior employees.

Superstars are passionate about their area of expertise. And if your candidate is passionate, then the hiring decision should be obvious. Still in doubt? Help me out Joel:

Never say “Maybe, I can’t tell.” If you can’t tell, that means No Hire. It’s really easier than you’d think. Can’t tell? Just say no! If you are on the fence, that means No Hire. Never say, “Well, Hire, I guess, but I’m a little bit concerned about…” That’s a No Hire as well. Mechanically translate all the waffling to “no” and you’ll be all right.

Why am I so hard-nosed about this? It’s because it is much, much better to reject a good candidate than to accept a bad candidate. A bad candidate will cost a lot of money and effort and waste other people’s time ...

5. Specialists do hard things and great things

By the time that you interview (or meet) your candidate, they (whether junior or not) will have already accomplished something demonstrably hard. And hopefully something demonstrably great.

Your candidate will already be an expert. Your candidate will already be a specialist. Your candidate will already have been accepted into (and completed) a competitive university course. Or they will have built their own rocket ship. Or whatever.

Oh … and … er … sorry … what do you mean:

Why would I want to hire a superstar?

And:

Won’t they make my life difficult?

Ah, I’m sorry. I can’t help you there. But now that you mention it, maybe I should write another article about how to identify superstar employers …

This post originally appeared at iainmclaren.com. These opinions are mine. They are not necessarily those of my employer.

Why are manufacturers and retailers more efficient than services companies?

Wed, 17 Sep 2014 00:00:00 +0000

The former win or die based on the efficiency of their supply chains

1. We can divide all industries into two broad categories

An executive friend of mine (who I will not name for reasons that will soon become clear) recently moved from a global energy company to a global services company. This executive was shocked that the services company’s operations and procurement processes lack focus and are very ad-hoc. This lack of operational (and procurement) focus is in stark contrast to the executive’s previous experience at the energy company.

This prompted a great discussion that I think is worth sharing with you.

Have you noticed that the car industry, large shopping centres, and transport companies have incredibly sophisticated operational (and procurement) processes, but global services companies often do not? This is because all industries can be split into one of two groups:

organisations that either win or die based on the efficiency of their supply chains; and
organisations that do not, and therefore focus (hopefully relentlessly) on the cost and quality of their people and technology.

Table 1 - Cost and efficiency focus by industry

Industry (NAICS designation) [1]	Industry type (NAICS designation) [2]	Focus
Natural Resources and Mining	Goods producing	Supply chain
Construction	Goods producing	Supply chain
Manufacturing	Goods producing	Supply chain
Trade, Transportation, and Utilities	Service providing	Supply chain
Information	Service providing	People and technology
Financial Activities	Service providing	People and technology
Professional and Business Services	Service providing	People and technology
Education and Health Services	Service providing	People and technology
Leisure and Hospitality	Service providing	People and technology
Other Services (except Public Administration)	Service providing	People and technology

2. Organisations with a supply chain focus are usually incredibly sophisticated

The car industry is famous for the sophistication of its supply chain and procurement methodologies. But the car industry is not alone – all organisations in industries, that live or die based on the efficiency of their supply chains, only survive if they have world class operations, and world class procurement functions.

That is why companies that focus on supply chains constantly and continuously improve their operations and procurement processes. These companies relentlessly focus on reducing their supply chain costs and increasing their supply chain efficiency.

And the process often begins when a CEO (or CFO or COO) tells her 2IC to reduce the company’s supply chain costs by 10% while (of course) increasing operational efficiencies.

Then the fun begins.

3. Organisations with a people and technology focus are often not quite so sophisticated

This is why executives (such as my friend) are surprised when they move from companies in industries that are supply chain focused to companies in industries that are not (i.e. when they move to companies in industries that are instead focused on improving their processes in relation to people and technology). These executives are surprised by their new company’s (often total) lack of operational and procurement sophistication.

Therefore, if our business is in an industry that does not live or die based on supply chain efficiencies, then our business will tend to have much less sophisticated operations and procurement groups (and processes) than businesses in industries that do win or die based on the efficiency of their supply chains.

And focusing on people and technology (rather than the supply chain) requires a nuanced industry and company specific approach. There is something to be said for the simplicity of simply being required to squeeze supply chains. Even if the process of actually squeezing these supply chains is extremely hard to do well.

4. How to build your business

So my friend’s task is to build world-class operational and procurement processes for a business, that is in an industry, that does not win or die based on the efficiency of its supply chains.

And while all companies are (obviously) different, we can use one (or more) of the innumerable established business process improvement methodologies when we focus on improving supply chains.

But industries that instead focus on improvements in relation to their people and technology (and do not win or die based on the efficiency of their supply chains) require a different approach. And possibly (and definitely arguably) a more nuanced approach.

“So it’s easy.” I said. And the executive laughed.

Footnotes

[1] North American Industry Classification System (NAICS) industries and industry groups listed on the the United States Bureau of Labor Statistics website.

[2] Ibid.

This post originally appeared at iainmclaren.com. These opinions are mine. They are not necessarily those of my employer.

It's time to face it - printing is ridiculous

Wed, 10 Sep 2014 00:00:00 +0000

Forget saving the environment. Printing at work is a ridiculous waste of time.

1. Printing at work is ludicrous

A few months ago, I needed to print and sign forms for our internal electronic expenses system. And yes, I love irony. But I haven’t printed at work since.

I don’t print because this significantly increases my personal productivity. Don’t get me wrong. I love paper and books. Lemony Snicket[1] said it best:

Never trust anyone who has not brought a book with them.

2. And handwritten work notes are ridiculous

But I sat in a meeting (last week) that involved a lot of very serious people scribbling furiously on legal pads. And I thought - what an incredible waste of time.

Because here’s the thing. It’s not real if it’s not stored in electronic form. If it’s important, we type that information into an electronic device. We avoid wasting time typing our hand-written notes.

So let’s not write things down. Let’s just type our meeting notes into word documents. Or whatever application suits our workflow - there is no law against just typing an email to yourself that contains your meeting notes.[2]

Whatever works.

3. Wireless technology required

Why is going paperless possible now? Always-on wireless technology. The ability to pick up our laptops, and take them with us to any meeting anywhere. Your organisation does let you do that, and provides you with global wireless internet access (and backups). Doesn’t it?

And yes, I find that I also need a smart phone for tasks, and a tablet as a replacement whiteboard (or to add a copy of my signature to a pdf document if required). But that’s it.

4. But I need to print large documents! Fallacy.

I am a lawyer, so I printed work documents on an enormous scale for over a decade. For example, when I moved law firms, I archived walls and walls full of floor to ceiling bookcases full of printed files.

Don’t get me wrong. I am not proud of destroying rainforests. But my point is that choosing not to print is a productivity issue. Not an environmental issue.

Put simply, I am more effective if I don’t print, and there is no need to print to be effective. Even when dealing with huge numbers of huge documents.

5. And I need to hand write notes, and print to double check my work! Also a fallacy.

Lawyers trained at large firms always double (or triple, or quadruple) check their work. For example, I was trained to:

print out a Microsoft Word document;
hand write changes to the document;
ask my secretary to type those changes into the Microsoft Word document (while always tracking changes);
email the document to the client (and the other parties involved in the transaction); and
finally, print out the document (and the related email) to go on the file.

But we achieve the same goal (i.e. spot typos, errors, and outright mistakes and gaps in logic) if we make our changes to an electronic copy of the applicable document. And we can then move on to something else (preferably overnight), and then re-check the document one more time (preferably the next morning) before we send it out.

6. Don’t print. You will never go back.

I stopped printing documents and, after a month or two, did not miss having paper copies of those documents. Even for enormous documents with multiple attachments.

I do have a soft spot for books and the antiquated. Really. For example, Moleskin notebooks are beautiful. And I have friends who write notes and diagrams in these notebooks, and then use their phone cameras to digitise those written notes. But I think that it’s time to recognise that printing at work (and possibly even writing pen and paper work notes) is a ridiculous waste of time.

So I challenge you to not print for three months. And if you are bold, I also challenge you to refrain from writing with pen and paper during this three month period.

Oh, and yes, it would also be great if we cut down fewer trees.

Footnotes

[1] And yes Josh, I did just quote Lemony Snicket (from Horseradish). That’s $5 you owe me.

[2] Well, actually, there are some laws that apply in relation to regulated industries (for example) that restrict the transmission of highly sensitive information over the internet (including via email). But these laws only require us to change our workflow (i.e. potentially change which apps we use to type in notes). But these laws do not change the general principle - that we can type instead of write on paper.

This post originally appeared at iainmclaren.com. Thanks to Josh Morris for reviewing an early draft. These opinions are mine. They are not necessarily those of my employer. And this post is not legal advice. Seek it if you need it.

Think before you upload

Thu, 04 Sep 2014 00:00:00 +0000

Control what you create

1. Control what you create

Soon after very private celebrity photos were released to the public, Ricky Gervais posted (and then swiftly deleted) the following note:

Celebrities, make it harder for people to get nude pics of you from your computer by not putting nude pics of yourself on your computer.

It was an awful, tone deaf, thing to say. And the victims are not at fault. But it actually crystallised my thinking about how we can, and in fact need to, control what we create.

2. Make hard choices about what you create

Posting these very personal photos on the internet is an ugly demonstration of why we need to actively control our:

private personal information (i.e. photos, emails, personal and work documents, tax information, etc.); and
public information (i.e. everything that we post on “social media”).

But our choices are not simple. For private information, we need to choose between privacy and backups. And for public information, we need to choose between control and convenience.

3. Private information: choose between privacy and backups

Chris Leishmann used to share a study with a friend of his at College. Chris and his friend would try to break into each other’s running computer (server). I didn’t know Chris’ friend, but Chris is an extraordinarily clever technology professional, and he could not figure out a way to stop his friend from breaking into his running server.

I recently discussed this issue with some very smart Amazon AWS security professionals. I won’t name them because I promised not to. But they confirmed that there is no such thing as a (commercially available) encrypted operating system. This means that your (malicious) cloud provider will always have access to all of your data, even if you encrypt the data (unless your cloud provider just provides “dumb” storage of your encrypted files, and does not process those files).[1]

Therefore, because phones and computers always break eventually, our choice is to either:

send our information (photos, files, or whatever) to a cloud provider and risk that information getting out into the wild; or
not send our information to a cloud provider, and risk losing all of our data when our phones or computers break (or if we lose the phone or computer).

Nasty.

Up until recently, most people just chose backups over privacy. Choosing to backup (and choosing to backup is almost always the right choice) is based on the assumption that your cloud provider’s host systems are secure. And based on the assumption that your cloud provider is not malicious. But as Ricky points out, the downside of trusting cloud providers is now clearly apparent.

It is unclear who first said that if your internet service is free then you are the product and not the customer (it was certainly mentioned on Bruce Schneier’s website in 2010). And if a service is free, then your social media provider will use your personal information to make money, for example by selling your personal information to advertisers.

This is not a bad thing. Useful social networks (like Linkedin) would not exist without a critical mass of users providing personal information. But social media, and free services in particular, come at a price (Community Services - Waffle):

The reason I don’t like social media is that it takes two things that are polar opposites and duct tapes them together. Your own utility – to save links, to write text, to move files or materials, to keep notes, to communicate with yourself in the future, to communicate with some other specific people – and the social media outlet’s desire to fulfil its own objectives first.

This is a recipe for tone-deafness at best. But it’s also an explanation of why so many people are so uneasy with social media ... Social media has come to symbolise, for me, ... the inability to define my own boundaries and the uncertainty about what’s going to happen tomorrow to the fundamental structure of this tool that I’m using – all the while someone ... makes money off of me.

If Facebook, or Twitter, or Linkedin, or your free email provider, disappeared tomorrow, or decided to remove or hide some or all of your data, will you lose any irreplaceable comments or photos or emails?

We benefit from the convenience of social networks. But we lose control of our personal information.

5. Sometimes choose not to store information at all

It’s sad. But we now need to assume that any device (e.g. computer or phone) that is connected to the internet can be broken into (i.e. cracked).

So it is not fair to say. And it is not fair that it’s true. But if we are about to create something that will cause us immense pain if posted on the internet. Be it naked photos, or emails that we would rather no-one ever see. Then we should consider not creating the thing that will cause us pain in the first place.

It’s ugly. I’m sorry. But please don’t shoot the messenger.

Footnotes

[1] Even if our data is encrypted on cloud servers, malicious cloud providers can still monitor our running operating system, and access any data processed by that operating system. And the only way to ensure that your cloud provider cannot access your data is to encrypt the data at your end (i.e. before you send it to the cloud provider). But cloud services can only work that way if your cloud provider just provides “dumb” storage of your encrypted files, and does not access, process, or use your information in any way. So cloud services rarely work that way.

This post originally appeared at iainmclaren.com. Thanks to Elaine Bevington and Josh Morris for reviewing early drafts. These opinions are mine. They are not necessarily those of my employer.

Robin Williams (1951 - 2014)

Tue, 12 Aug 2014 00:00:00 +0000

He was a giant

1. He was a genius

How do we start. Even start. To get to grips with how important Robin Williams was to my generation. How can I even start to explain how important he was to me?

I don’t think it’s possible. It’s like trying to explain the importance of family. My friends used to make fun of me by pointing out that I have two types of laugh. One is an everyday laugh. The way that I laugh when a colleague tells a joke. A wry, amused laugh. It’s real, but it’s the light milk of laughs.

Robin Williams always induced the other type of laugh. That deep, intensely personal, belly laugh. The laugh in spite of ourselves. The laugh that children laugh. His genius was that he didn’t need to shock or offend. He could, without fail, make us laugh out loud using nothing but family friendly material and celebrity impressions. He was so funny. He was so integral to what we thought of as being genuinely funny that he became the epitome of funny. He became a part of who we are. He is a part of who we are.

Attempting to explain his importance is like trying to explain how important Derek Jeter is to someone who has never seen a game of baseball. Nike nailed it. Watch that video. Please. And then come back.

2. There is no way that I’m going to the movies to see a stupid cartoon

The year was 1992. I was in year 10 (the third last year of school). Some friends of a girl who I was sort of seeing at the time (I had made the whole thing unnecessarily complicated) suggested that we all go to the movies to see Aladdin. Long story short, she was wonderful and I was very very stupid. In my defence, I was only, what, 16, at the time? But that is no excuse. I completely stuffed the whole thing up. If she is reading this, then I really am truly sorry. I only have a few genuine regrets in life and that is one of them.

But anyway. This was before all of that. Her friends suggested, and then insisted with ever-increasing intensity, that we all go to the movies to see Aladdin.

For the youngsters out there, it’s important to remember that this was the first. The very first. Animated movie that was funny for both adults and children. I was absolutely convinced that it would be an excruciating waste of time. And even the prospect of going to the movies with this very nice girl was not enough of an incentive to convince me to go.

3. His genius was that he made it look effortless

Boy was I stupid. I never did see Aladdin at the movies, and it is one of the funniest movies that I have ever seen. What makes it so great is that you get the impression that the producer and director put Robin Williams in front of a microphone, and said:

You are the Genie in 1001 Nights. Let’s record your dialog, and then we will just build the movie around the recording. We will add the other voices, the animations, and the actual story later. Go!

I mean, he used to go on talk shows, and the host would basically say the name of one celebrity after another, and off he would go. Now perhaps he spent the rest of his life preparing material so that he could be ready for these “be funny - go!” requests. But I don’t think so. He was just too quick. And just too immediately and instantly funny. He was obviously doing bits on the spot. Bits that other comedians would spend months preparing and pouring over. Bits that other comedians would kill for.

4. Saying thanks seems like nothing at all

I can’t bring myself to link to any of his clips and movies. Well OK, just one. His recent hilarious impression of what a stereotypical French phone (French Siri) might say if you asked it for a restaurant recommendation in France.

You’re in France! Walk outside! Walk down a block! Look around you idiot! What are you stupid?! … Look around you why must you ask a phone? Live your life! Taking pictures with your phone?! Look! Look. And then paint!

So instead I will link to this video of what Keith Olbermann said when David Letterman retired. An ode to another genius comedian:

For all the other days I just stared and laughed at how many times better he was than me or anybody else I’d ever seen. Babe Ruth announced he’s retiring today.

Thanks Robin. Just thanks.

This post originally appeared at iainmclaren.com.

The Two Giants - the Amazon and Google cloud problem

Tue, 05 Aug 2014 00:00:00 +0000

Amazon and Google run our critical cloud infrastructure but can switch it off at any time. What’s your plan when the cloud goes down?

1. The cloud used to be called floppy disks

We need the cloud.

Back in the stone ages (i.e. when I attended University), I worked at the University information technology services department. Our brief was to solve any and all technology problems of our University staff members and students. And their friends and family. Basically, we helped anyone who asked.

It was both stupid and glorious. We fixed printers. We helped out with crazy programming assignments. We diagnosed weird email problems over the phone. We fixed Microsoft Word formatting, and helped with Excel formulas. We pulled computers apart and put them together again. We explained how email worked. And what this new “internet” thing is. Yes I am old.

But the floppy disks. That is what stuck with me. In those days, if you wanted to read email, you installed an email program on a floppy disk. And all your emails were also stored on that floppy disk. I bet you can see where this story is going …

Three times during my stint in information technology services, PhD candidates handed me a floppy disk containing the only copy of their PhD thesis. And the floppy disk was broken. Each time I managed to retrieve the entire file, or at least the text of the thesis. But for many other students and staff who came to us with broken floppy disks, their assignments were irretrievably lost.

2. Don’t run your own personal cloud

It was around this time that we (or at least I - maybe I am slow) realised that we need to store our personal electronic files securely. And the same logic applies to photos now that we have switched from film to digital photos, and now that we take many many more photos.

It is actually very difficult to securely store our files. Let’s use email as an example. email is the most important internet application. Do you host your own email server? I used to be responsible for running email servers for 100+ users and let me tell you, even at that small scale, it is very very stressful worrying about 24x7x365 uptime. And you will never ever be forgiven if you lose the emails of your users.

Therefore, these days we, quite sensibly, let technology companies run our mail severs for us. Doing it yourself is dangerous and therefore not realistic.

3. So we rely on cloud providers … that can switch off the cloud at any time

Google (gmail) and Amazon (aws) can switch off these cloud services at any time. I am not saying that they are malicious. Quite the contrary. Both are fantastic companies. And we need their services. But both companies deliberately and quite sensibly (for them) maintain the flexibility to “decommission” technology services whenever they want.

Because if you live in the western world, then your personal email is probably stored by Google (or one of its competitors). And one way or another, everyone relies on Amazon. As Benedict Evans says (in Losing my Amazon Religion):

And to be sure, Amazon has had doubters from the beginning: few gave the company a chance when it started, and even fewer thought it would survive the bursting of the bubble. And yet, today Amazon is the king of e-commerce, at least in the United States, and the clear leader in cloud services, and now they are making a big push into digital content and devices.

Amazon has an enormous global fleet of data centers, and supplies infrastructure that is critical for:

a significant proportion of large companies (in the western world at least); and
the suppliers of these large companies.

For example, Amazon runs the servers used for everything from online storage services to services used by governments, financial institutions and airlines.

4. What can we do?

There is nothing we can do. At least about companies that we rely on but do not control. If those companies are not prepared for the day that Amazon and Google switch off their critical cloud services.

However, we can do two things to protect ourselves (and our companies) from the disappearing cloud:

Have a plan if the cloud service or company disappears overnight. For example, can another supplier provide the same service? Or can we run the service in-house?
Keep up-to-date copies of all of our data that is stored by our cloud providers. Preferably on our own physical infrastructure (or at least with an independent third party).

While these requirements may seem simple, they are often complicated to implement, particularly for large companies.

But don’t blame the giants. They are happy to help. Just don’t count on them sticking around forever.

This post originally appeared at iainmclaren.com. Thanks to Josh Morris for reviewing an early draft. These opinions are mine. They are not necessarily those of my employer.

Perfection is worth risking your company and your job

Wed, 30 Jul 2014 00:00:00 +0000

Fight the culture of cutting corners.

1. Perfection is always in fashion

We are paid to get it right. That is how I was trained. Typos were not tolerated. Never-mind out-and-out mistakes. It was built into our organisational culture. But we live in a world of risk and it is surprisingly easy to let mistakes slide. At least for a while. And as Peter Drucker (famously allegedly) once said:

culture eats strategy for breakfast.

“Risk management”. That’s the euphemism that we use at work when we want to do the wrong thing. Or the easy thing. When the hard thing is the right thing. I may be an outlier on this issue. But I think that true risk management starts with the perfect solution. And we should only accept (minor changes to) perfection. We should not accept second best. Or that our only option is to accept the average.

It doesn’t matter the industry. For me, it is a contract for a major project where I would not change a single word in a single clause. Or a software program when I know. Just know. That I won’t change it no matter how long I stare at the lines and lines of code.

2. Compromise means do it almost perfectly. Not do it wrong.

“Boiling the ocean”. That’s how we criticise our colleagues for taking the time to do it right. And of course, of course, we should consider reasonable compromises to reduce costs, and rapidly solve problems. But as Shawn Blank says:

Average, “good enough” work is no longer good enough. Our success, our products, and our reputations all rest on the details, the delight, the intention, and the vision we bring to our work.

If I start with the easy, wrong, answer. Or even the assumption of failure. Then I don’t learn. But worse than that, the problem always returns. Eventually and magnified.

3. Embrace perfection

Insisting on perfection at work is very hard in practice. And I realise that I may be in the minority here. But as Bruce Schneier says:

[o]f course, it’s actually more complicated than that. A person might decide to break the norms … because his moral compass tells him to … [and] sometimes we decide a norm breaker did the right thing.

Let’s stop using “risk management” as code for hiding the broken. Or accepting poor work (or even failure) as inevitable. Because the greatest risk to our companies and our jobs is the embrace of mediocrity.

Instead, let’s risk striving for (minor deviations from) perfection.

This post originally appeared at iainmclaren.com. Thanks to Josh Morris for reviewing an early draft. These opinions are mine. They are not necessarily those of my employer.

Are banks doomed?

Wed, 23 Jul 2014 00:00:00 +0000

Telecommunications services were commoditised in the early 2000s. Are banks next?

1. The rise and fall of the telcos

Ben Evans recently made the point (in Unbundling innovation: Samsung, PCs and China) that most truly innovative internet services are invented by companies that are built (from the ground up) to sell technology to third parties. In other words, by companies that sell technology services as their core business.

But what if the CEO’s elevator pitch does not include the words “sell technology services”, but instead includes the words “sell telecommunications services”? Or for that matter, “sell banking services”. Will that company be be able to invent and sell world leading “digital” services? Before the iPhone was released (i.e. back in the early 2000s), telecommunications companies certainly thought so.

As Ben says, the telcos were …

“intensely aware of all the cool stuff that was going to happen with mobile and the internet. They predicted a great deal of it very accurately, but they thought that they would be doing all of it”.

They thought wrong.

Traditionally, telcos controlled the services that run on their physical infrastructure (i.e. over their land lines and fiber networks, and via their wireless spectrum). The telcos were able to book significant margins for these services (e.g. phone minutes and SMS). Telcos also tightly controlled the types of phones (and other similar devices) that customers could buy.

But, as Ben points out, the telcos were unable to compete (i.e. maintain control over these devices and services) because:

they were structurally bad at making services;
even if they were good, those services were just one amongst many; and
the network effects for these services ran across the whole internet, not just their customers.

In other words, the iPhone and then Android ate their collective lunch.

Specialist technology service providers sold (i.e. unbundled) physical phones and services from the infrastructure owned by the telcos. And you have to think that the telcos will never again make enormous margins on these services (e.g. phone minutes and SMS).

2. Will banking services also be commoditised?

This brings us to banks.

McKinsey also published an article recently about the rise of the digital bank. The argument in this article is that as “European consumers move online, retail banks will have to follow. The problem is that most banks aren’t ready”.

I am sure that this is true, but I think that the article misses the wider point. We should instead be asking whether banks are failing to provide “digital” services because banks are not ready, or because banks cannot (i.e. are not built to) compete with dedicated technology service providers.

Let’s take a step back. What services do banks actually provide? For most individual customers, banks (in Western countries at least):

store their money in savings accounts;
allow customers to pay for goods and services using debit and credit cards;
allow customers to withdraw cash from ATMs;
provide internet banking services (via the web and mobile apps) to allow customers to check their account balances and transfer money; and
loan money to these customers (for houses and other tangible goods).

Banks also provide lots of other services. Including, importantly, liquidity to businesses and individuals. And I am sure that I have missed other key services. But I think that the above gives us a general idea of the core reasons why most individual customers use banks. For example, customers can also:

go into a branch (although, unless there is some sort of problem, how often do they do this?);
pay for goods and services using cheques (a payment method that cannot be much longer for this world);
provide “wealth” services, superannuation and insurance advice;
etc.

Therefore the telcos may be in a better position than the banks. Telcos have infrastructure (i.e. the landlines, fibre networks, and wireless spectrum) that may become, or arguably already are, commodities. But people still need this telco infrastructure to actually use telecommunications and internet services (i.e. to make phone calls and access the internet).

On the other hand, unlike telcos, if all of the bank “infrastructure” (other than possibly ATMs) dissapeared, in the long term how much would that effect the average banking customer? And do these current bank customers actually need to deal with banks? Especially if bank competitors (e.g bitcoin or paypal) provide ways to store money, pay for goods and services (removing the requirement for cards, ATMs, and physical cash), and loan money to customers at scale (e.g. and don’t laugh, by croudsourcing).

3. Are banks doomed?

No. I think. But here is Ben again:

The analogy I often use in this case is that [this] ... is like a municipal water company deciding to get into the soda business - because it knows water, and has trucks, and customers trust its brand. Even if it managed to come up with a great soda, it would still be just another can of soda amongst many.  

It isn't their place to provide it ... even if they could.

To win, banks need to stay ahead of their non-bank competitors. And provide world-beating, or at least world leading, technology services. Particularly payment services.

That is the challenge.

This post originally appeared at iainmclaren.com. Thanks to Josh Morris for reviewing an early draft. These opinions are mine. They are not necessarily those of my employer. Or Josh.

8 rules for closing contracts

Wed, 16 Jul 2014 00:00:00 +0000

“Amateurs discuss strategy. Dilettantes discuss tactics. Professionals discuss logistics.” - Unknown

Yes, we can close any1 contract by following eight2 simple rules. Two weeks ago:

Client: "No, we are not doing that.  We  need to follow Iain's rules."  

Me, incredulously: "My what?"

Then it hit me. There are actually eight simple rules. Or actually steps. That we can follow if we want to quickly close contracts. And then another client asked me to create a “Markup and Document Exchange Protocol” or a “simple document to accompany initial drafts of agreement documents to establish … preferred protocols.”

My clients are the best. Really. But that sounds awful. I mean really awful. Worse than doing my taxes awful. So instead, I propose that we follow eight simple and practical steps to close contracts:

1. The customer and the supplier meet in person

Or at least via teleconference. To ensure … or let’s be realistic at least significantly increase the likelihood … that we all … at least broadly … understand the scope of the services that will be provided by the supplier to the customer.

I know I know. It seems obvious. But skipping this step is a killer. It is amazing how often that we find that what the supplier thinks it is selling is very different to what the customer thinks it is buying.

2. The customer provides the first draft of the contract to the supplier

The supplier does not provide the first draft. Yes, I said it.3

And if you are a customer and you do not have an appropriate template? Ask a law firm, that specialises in what we are doing, to draft one for you. Oh, and in general, with law firms, we get what we pay for. And … ah … no …. I’m getting distracted. Engaging great lawyers is definitely a discussion for another day …

3. The supplier reviews the contract and then phones the customer to complain

I am being slightly. Just slightly. Tongue in cheek here. What I mean is that the supplier should only make this call to the customer if the agreement is … just … completely and utterly … wrong.

If the supplier makes this call and the customer is convinced, then we return to step (2). And yes, this document is turning into “Choose your own Adventure”. And yes, if you have no idea what “Choose your own Adventure” is, then you are just too too young … and … where was I?

Oh yes …

4. The supplier marks up that first draft of the agreement

No Microsoft Word comments. No issues lists. Just turn on Microsoft Word track changes and mark-up that document.

Commit don’t preside.

Trust me when I say that our drafting changes will be self-explanatory. Well, almost always. And if we really feel that our drafting changes are not self-explanatory, then we can insert comments as tracked changes in square brackets. For example:

[Supplier note: We have deleted this clause requiring us to coat all of our elephants in solid gold. While we understand the customer’s preference for shiny things, this would result in a significant price uplift. And probably kill the elephants.]

And for what it’s worth, these comments really pop if you highlight them in yellow. I’m just saying. But anyway …

5. The customer reviews the supplier’s draft and then phones the supplier to complain

Tongues and cheeks again. At least a little. The customer only needs to make this phone call if the supplier has not followed the above process. Or if the supplier has “shredded” the contract. For example, by … oh, I don’t know … say … deleting large chunks of the customer’s drafting, and replacing that drafting with the supplier’s standard terms and conditions.

In other words, if we have not followed the process, then we return to step (4). Or if we have followed the process then …

6. The customer marks up the supplier’s draft of the agreement

All of the rules in step (4) apply. Accept all of the supplier’s changes, turn on Microsoft Word track changes, and just mark-up the document. We are trying to “close” the contract, so really try not to introduce new terms and conditions.

Seriously. Apologise to the supplier if we have to introduce whole new clauses. Our goal is to hold our ground, or compromise if we must. We will never finish if we keep adding new clauses.

7. Continue exchanging drafts

We will have built up some momentum now. We close by (in descending order of preference):

exchanging drafts as set out above (i.e. accept all changes, mark-up, and apologise if we add new clauses);
resolving issues during phone calls and/or teleconferences; and
attending face to face meetings.

The fewer meetings we have, the faster we close. Counterintuitive but true. I promise you. So at this stage emails are better than phone calls. And phone calls are better than face to face meetings.

Therefore:

It is best to exchange drafts when we are just dealing with “drafting” issues rather than any “commercial” or “legal” disagreements.
Then, hopefully, we will end up with a draft of the agreement with much fewer redlined changes. And we can run through the latest draft of the agreement during phone calls. Hold the line against issues lists.
Then, once we are left with, say, 5 to 10 issues, we can get in a room to resolve the final issues. For large agreements, at this stage we will probably involve C level executives.

Resist getting in a room for days at a time from the start. Unless we are into wasting our lives having fascinating meta-discussions regarding contracting principles. In other words, let’s just …

8. Close

Close. Close. Close. Apparently it is as simple as ABC …

Footnotes

[1]: And yes, many of the above rules break down for smaller or less important agreements. And drawing the line between “important” and “unimportant” contracts is tricky. The rules also change if the supplier has significant market power. Like Newtonian physics breaks down when we get quantum. Or appears to. Is it just me, or are theoretical physicists just inventing whole imaginary disciplines at this point? …

[2]: Because 7 (Samurai) was taken. And because 8 is lucky. And because proposing [x] rules for [y] things is fantastic clickbait. Apparently. And 8 seems real because it is not divisible by 5. Luck + clicks = profit!

[3]: There will be shouting. Mark my words.

This post originally appeared at iainmclaren.com. Thanks to Josh Morris for reviewing early drafts. These opinions are mine. They are not necessarily those of my employer. And this is not legal advice. Seek it if you need it.