Categories
code development programming

Software Development as a Christmas Story

Gingerbread Christmas Tree
Oh Tanenbaum

We decorated the house for Christmas. A smaller project than most of the software I’ve worked on, but a useful reflection.

The Cost of context switching

Sometimes, in the middle of one task, you need to do another. Either because you were interrupted, or because you weren’t prepared. Consider the tree, a tall tree, one you need a ladder to decorate. But you left the ๐ŸŒŸ in the attic.

You don’t just have the cost of picking up the star, you need to get down the ladder, fold it up, carry it upstairs, open the attic, unfold the ladder, climb to the attic, find the star, then pick it up and unwind all the other steps just to get back where you stated before you can continue the task of putting the Star on the tree.

It’s just a minute to get the star…

Yak shaving

Sometimes it’s more than just one thing. Sometimes to get to where you need to go, there’s a cascade of other tasks.

You want to put the tree up, but it’s the first time you’ve had a real tree, so you need a new base. It’s bigger than last year’s tree, so your lights don’t fit. So you need to go to the shops. But you need to fill up your car. And the tyre pressure warning light comes on so you need to top them up. And you need to check the tread depth whilst you’re there, and so on.

Programming in a nutshell

Performance at scale

Our tree stood up fine when it was delivered. But as it scaled out and the branches widened, it pushed against the wall, making it unstable in the condensed space. It fell over.

Luckily no lights or baubles were using it at the time, but it’s an interesting challenge holding up a heavy tree in one hand, trying to adjust it’s position with the other hand, as I avoided the puddles of water in the floor as my wife mopped them up. If you’ve ever worked support, this may sound familiar.

Turns out that it was harder to stabilise than I anticipated.

The branches were unevenly partitioned, providing more load on one side, so I had to stabilise it against the wall. And the tree was almost a foot taller than expected, which turned it out to be 2 foot taller than the base was rated for.

We upgraded the base to handle the load. It’s bigger than we need.

Technical debt

I have some old pre-LED tree lights and they slowly fail so each year I replace the unlit ones from the packet of spares but I haven’t been replacing the spares. Eventually, they will run out, and the spares will no longer be available. I’ll have to throw them all out and start again.

The new led ones don’t let me replace them individually. But they last longer and are cheaper to run.

Which debt is easier to live with?

With a big tree, those old lights aren’t long enough. So, do I buy another set for the rest of the tree that doesn’t match them, or throw out the existing ones and buy a longer set? The latter looks better, but throws out the sunk cost along with the lights.

You know computers, can you look at this for me?

No, I can’t fix your tree. I can navigate a garden centre. I know enough physics to keep a tree upright, eventually, and safely put the angel on, but I’ve no Idea how to grow one, or what that weird stand with 17 gears you bought does, or how to assemble your plastic one. And I’ve no idea what that bug in the tree is.

Merry Christmas. Catch you all in the New Year.

Categories
code development programming

How I approach coding challenges

This time last year I was doing a lots of tests for job interviews, and I was practicing with https://adventofcode.com

I know there’s various data structures and algorithms you can learn to make these easier (and I hope to come back to some of my favourites in a later post), but there’s a general approach I take to these that helps me, and might help some of you too.

Use TDD

I’ve always been a fan of Test-Driven Development, and these type of puzzles lend themselves well to TDD thinking. Often examples are provided that will form the initial test suite. The algorithm is often broken into steps (sometimes explicitly “On every clock tick…”)

Get into the habit of checking as you go to make the most of the information provided to you. Think about the problem first, and the mapping between the input and the output, before you write any code.

Check everything on simple solutions.

Have templates

Most challenges involve a few common tasks. Read input from a file (ignoring blank lines), create a list of…, output as CSV, etc.

Some of these will have standardised patterns in your chosen language, and you should use them. Some will likely require some error handling. Some will require loading from standard or 3rd party libraries. And all will require a test framework.

As you go, build up a utilities list of functions, classes and packages that you know how to use, are robust enough and have minimal clutter. They’re not meant to be usable by anyone but you, but you will use them a lot. Collect and cherish them.

Think simple

Technical tests and coding challenges, in general, are designed to have short, easy solutions. If you need to solve P == NP to make your code fast, it’s likely you’re using an algorithm that’s a poor fit. Find 2 other ways to solve the problem, and pick the simpler one.

Simpler than that

Think about the simplest solution that could possibly work and write it simply.
Make the code readable. This is not the time for clever solutions. When it fails on an input, make it easy to change.

But not that simple

You still need to think about edge cases, and memory usage, and a myriad of other resource constraints. Think about what could go wrong, within the confines of the puzzle. Advent of Code has a particular knack for detecting edge cases in your code for Part 2 of each day’s challenge.

Think fast

Beware of premature optimisation, but expect long inputs and many loops if you use the naive implementation. Learn new algorithms and data structures.

Categories
development programming

Somebody else’s problem

Sometimes dividing tasks creates inefficiency. When one person both cooks and cleans, they tend to create less washing up than if the tasks are divided, because washing becomes “somebody else’s problem”. However, when one person washes and the other dries, the load is equivalent. This setup can be prone to stoppages when the drainer is empty and the washer is trying to get that stubborn burnt bit off the pan, at which point any good ToC practitioner will ask the drier-upper to help with the washing to maintain throughput.

When teams are divided by layer instead of features, the interface between them becomes a fracture because each team has context missing from the other. Data has the wrong shape, validation requires fields that can’t be captured. If you build both sides, the transition is smooth.

When developers test, they can focus on smaller, independent chunks, which means fewer tests with greater coverage. Writing integration tests for complex calculations is painful and slow. Testing those same calculations as an xUnit Theory is cleaner and faster.

Specialists are great, but where is specialisation causing fractures and more work?

Categories
code development programming security

This is raw

This is raw chicken : ๐Ÿค

If you eat it like that, you may get hurt immediately, by its beak, or its claws. It may grab your money and run off with it.

If you want to eat it, better to kill it first. ๐Ÿ’€

If you eat it like that, you may get hurt or die, in a few hours, or days. Washing it won’t help.

Cook it. Cook it well. If there’s any sign of pink, cook it some more. ๐Ÿ”ฅ

It might still kill you, but at least you’re a lot safer than when you started.


This is raw data : ๐Ÿค

If you display it, users will get hurt immediately, whether by cross-site scripting, cookie sniffing, crypt-currency mining, or something else. If you’re lucky, it will be something your user’s see immediately and leave your site never to return. Otherwise they may get infected.

If you want to use it, better to validate it first. ๐Ÿ’€

If you save it like that, your users are still vulnerable. It might appear on the front end in a different form. It might be a string of unicode characters that crashes your phone. It might be a link to somewhere they can’t trust.

Encapsulate it. Sandbox it. Never trust it, in or out . HTML encode, whitelist the output as well as the input.

And if you need to avoid spam, or incitement, or solicitation, maybe you need editors. Computers can’t fix all the social problems. ๐Ÿ”ฅ

Categories
code development programming

Translations with React, Redux and Asp.Net

In a recent project, which was the first time I’ve used React and Redux in anger, we had a requirement to support 2 different languages, in the .Net backend for emails and PDFs, and in the React frontend.

As the translators we used were used to resx files, we wanted to use those as the master source. In other projects we’ve done the javascript translations via a pre-compilation step into static javascript files, but since redux has it’s own store, I decided to see if we could use that to store and process the translations.

This approach has the advantage that we can use the redux state to translate the site automatically when a new language is selected, without having to reload any pages. For an application that depends on up-to-date data, and had to operate on por data connections, avoiding reloads is essential.

The redux examples shown here use TypeScript, which certainly helped in our development process, but there’s a lot of gotchas getting the dotnet new react-redux template and many 3rd party react and redux libraries working nicely with TypeScript. You can make it work, but it’s definitely not an out-of-the-box solution in most cases.

The solution consists of 3 parts:

  1. The Asp.Net Core 2.0 controller to translate the resx data into JSON (in this example, the resource file is called UserFacingStrings.resx and the default is en-US,
  2. The Redux configuration to retrieve the data and populate it into the store; and
  3. A React function that reads the translation from the store and presents it to the user.

The solution below attempts to get the localisation from the user headers, so you’ll need to enable localisation in your Startup.cs, but depending on your use case, you may want save user language settings in the user’s account data, or in a cookie or other storage in their browser.

ASP.Net Core API controller

[Produces("application/json")]
[Route("api/Localisation")]
public class LocalisationController : Controller
{
    private static Dictionary _cultureCache = new Dictionary();

    /// Get all known translations
    /// Choose a culture (e.g. `en-US` )
    /// will default to browser culture if not specified
    [HttpGet("[action]/{culture?}")]
    public IActionResult Translations(string culture)
    {
        try
        {
            var cultureInfo = CultureInfo.CurrentUICulture;
            try
            {
                string preferredLanguage = HttpContext.GetPreferredLanguage();
                if (!string.IsNullOrWhiteSpace(culture))
                {
                    cultureInfo = CultureInfo.GetCultureInfo(culture);
                }
                else if (!string.IsNullOrWhiteSpace(preferredLanguage))
                {
                    cultureInfo = CultureInfo.GetCultureInfo(preferredLanguage);
                }
            }
            catch (CultureNotFoundException)
            {
                // Fallback to en-US
            }

            if (_cultureCache.Keys.Contains(cultureInfo.ToString()))
            {
                return new OkObjectResult(new { t = _cultureCache[cultureInfo.ToString()] });
            }

            var translations = new Dictionary();

            // Insert all resources, to then be overwritten
            if (cultureInfo.TwoLetterISOLanguageName != "en")
            {
                ExtractResources(translations, UserFacingStrings.ResourceManager.GetResourceSet(CultureInfo.GetCultureInfo("en-US"), true, true));
            }

            ExtractResources(translations, UserFacingStrings.ResourceManager.GetResourceSet(cultureInfo, true, true));

            _cultureCache[cultureInfo.ToString()] = translations;

            return new OkObjectResult(new { t = translations });
    }
    catch (Exception)
    {
        return NotFound();
    }
}

private static void ExtractResources(Dictionary translations, ResourceSet resourceSet)
{
    foreach (DictionaryEntry res in resourceSet)
    {
        translations[res.Key.ToString()] = res.Value.ToString();
    }
}

Browser Extension

public static class BrowserDetailsExtensions
{
    public static string GetPreferredLanguage(this HttpContext http)
    {
        return http?.Request
            ?.Headers["Accept-Language"].ToString()
            ?.Split(',').FirstOrDefault()
            ?.Split(';').FirstOrDefault();
    }
}

Redux plumbing

CultureActions.ts

import { AppThunkAction } from '../store/index';
import { fetch } from 'domain-task';
import { CALL_API } from 'redux-api-middleware';
import { actionCreatorFactory } from 'typescript-fsa';
import { Translations } from '../reducers/CultureState';
import { Dictionary } from 'lodash';

const API_ADDRESS: string = "/api/Localisation/";
const API_GET_TRANSLATIONS: string = "Translations/";
const API_LANGUAGE_CODE: string = ""; // "pt-BR", "en-GB"
const API_REQUEST_TYPE_GET: string = "GET";

export interface ApiError {
    name: 'ApiError',
    status: number,
    statusText: string,
    response: string
}

export interface GetTranslationsAction {
    type: 'RECEIVE_TRANSLATIONS';
    t: Dictionary;
};

export interface TranslationsApiPosted {
    type: 'Translations_REQUEST_POSTED';
    payload: never;
};

export interface TranslationsApiFailed {
    type: 'Translations_FAILURE';
    payload: ApiError;
    error: true;
};

export interface TranslationsApiSuccess {
    type: 'Translations_SUCCESS';
    t: Dictionary;
}

// Declare a 'discriminated union' type. This guarantees that all references to 'type' properties contain one of the
// declared type strings (and not any other arbitrary string).
export type CultureAction = GetTranslationsAction;

const actionCreator = actionCreatorFactory();
export const translationsAwaitingResponse = actionCreator('Translations_REQUEST_POSTED');
export const translationsSuccess = actionCreator('Translations_SUCCESS');
export const translationsFailure = actionCreator('Translations_FAILURE');

// ACTION CREATORS don't directly mutate state, but they can have external side-effects (such as loading data).
export const cultureActions = {
    getTranslations: (): AppThunkAction => (dispatch, getState) => {
        console.log("REQUEST TRANSLATIONS ACTION")
        const rsaaRequestAuctions = {
            [CALL_API]: {
                credentials: 'same-origin',
                endpoint: API_ADDRESS + API_GET_TRANSLATIONS + API_LANGUAGE_CODE,
                method: API_REQUEST_TYPE_GET,
                types: ['Translations_REQUEST_POSTED', 'Translations_SUCCESS', 'Translations_FAILURE']
            }
        }
        dispatch(rsaaRequestAuctions);
    },
};

CultureStore.ts and the tr() method

import { Dictionary } from "lodash";
import * as He from 'he';
import { Decimal } from "decimal.js";

export type CultureState = {
    strings: Translations
};

export interface Translations {
    t: Dictionary
};

export const emptyCulture: CultureState = {
    strings: {
        t: {}
    }
};

export function tr(culture: CultureState, key: string): string {
    try {
        return He.decode(culture.strings.t[key]) || key;
    } catch (error) {
        console.log("translation for '" + key + "' not found");
        if (key == undefined || key == null) {
            return "- -";
        }
        return "-".repeat(key.length);
    }
}

// Returns the key value itself if no match is found in the resx
export function trFallback(culture: CultureState, key: string): string {
    try {
        return He.decode(culture.strings.t[key]) || key;
    } catch (error) {
        return key;
    }
}

export function trFormat(culture: CultureState, key: string, args: string[]): string {
    return formatString(tr(culture, key), args);
}

export function formatDecimalAmount(subject: Decimal): string {
    return formatAmount(subject.toNumber());
};

export function formatAmount(subject: number): string {
    return subject.toLocaleString(navigator.language, { maximumFractionDigits: 2 });
};

function formatString(subject: string, args: string[]): string {
    if (subject === undefined) {
        return "";
    }
    return subject.replace(/{(\d+)}/g, function (match, number) {
        return typeof args[number] != 'undefined'
            ? args[number]
            : match
        ;
     });
};

CultureReducer.ts

import { Action, Reducer } from 'redux';
import { isType } from 'typescript-fsa';
import { Translations, emptyCulture, CultureState } from './CultureState';
import { ApiError, CultureAction, translationsAwaitingResponse, translationsFailure, translationsSuccess } from '../actions/CultureActions';
import { Dictionary } from 'lodash';

export interface CultureStateR extends CultureState { }

const unloadedState: CultureStateR = emptyCulture;

export const reducer: Reducer = (state: CultureStateR, incomingAction: Action) => {
    const action = incomingAction as CultureAction;

    if (typeof state === undefined) {
        return unloadedState;
    }

    if (isType(incomingAction, translationsAwaitingResponse)) {
        return {
            strings: state.strings
        };
    }
    if (isType(incomingAction, translationsFailure)) {
        return {
            strings: state.strings
        };
    }
    if (isType(incomingAction, translationsSuccess)) {
        const rawTranslations: Dictionary = incomingAction.payload.t;
        return {
            strings: { t: rawTranslations }
        };
    }

    switch (action.type) {
        case 'RECEIVE_TRANSLATIONS':
            return state;

        default:
            break;
    }

    return state || unloadedState;
}

Use in account controller

import { CultureState, tr } from '../reducers/CultureState';
/// Some imports removed for clarity

/// Add CultureState into props for this component
type AccountProps = AccountState & CultureState & typeof accountActions & RouteComponentProps;

export class Account extends React.Component {
}

Use tr() in view tsx file

</pre>
<h2>{tr(this.props, "PersonalDetails")}</h2>
<table>
<tbody>
<tr>
<td>{tr(this.props, "Name")}:</td>
<td>{this.props.user.userName}</td>
<td><a href="/account/profile">{tr(this.props, "Edit")}</a></td>
</tr>
<tr>
<td>{tr(this.props, "Company")}:</td>
<td>{this.props.user.companyName}</td>
</tr>
<tr>
<td>{tr(this.props, "Email")}:</td>
<td>{this.props.user.emailAddress}</td>
</tr>
</tbody>
</table>
<pre>
Categories
development programming security

NMandelbrot : running arbitrary code on client

As part of my grand plan for map-reduce in JavaScript and zero-install distributed computing, I had to think about how to gain user trust in a security context where we don’t trust the server. I couldn’t come up with a good answer.

Since then, we’ve seen stories of malicious JavaScript installed to mine cryptocurrenciesย , we know that JavaScript can be exploited to read kernel memory, including passwords, on the client, and I suspect we’ll see a lot more restrictions on what JavaScript is allowed to do – although as the Spectre exploit is fundamentally an array read, it’s going to be a complex fix at multiple levels.

I had ideas about how to sandbox the client JavaScript (I was looking at Python’s virtualenv and Docker containers to isolate code, as well as locking them into service workers which already have a vastly more limited API), but that relies on the browser and OS maintaining separation, and if VMs can’t maintain separationย with their levels of isolation, it’s not an easy job for browser developers, or anyone running on their platform.

I also wondered if the clients should be written in a functional language that transpiled to JavaScript, to have language level enforcement of immutability and safety checks. And of course, because a functional style and API provides a simpler context to reason about map-reduce, by avoiding any implicit shared context.

Do you allow someone else’s JavaScript on your site, whether a library, or a tracking script, or random ads from Russia, Korea, botnets and script kiddies? How do you keep your customers safe? And how do you isolate processes you trust from processes that deal with the outside world and users? JavaScript will be more secure in the future, and the research is fascinating (JavaScript Zero: real JavaScript, and zero side-channel attacks) but can you afford to wait?

Meltdown and Spectre shouldn’t change any of this. But now is a good time to think about it. Make 2018 the year you become paranoid about users, 3rd parties and other threats. The year is still young, but the exploits are piling up.

 

Categories
development programming

Cloud thinking : storage as data structures

We’ve all experienced the performance implications of saving files. We use buffers, and we use them asynchronously.

Azure storage has Blobs. They look like files, and they fail like files if you write a line at a time rather than buffering. But they’re not files, they’re data structures, and you need to understand them as you need to understand O(N) when looking at Arrays and Linked Lists. Are you optimising for inserts, appends or reads, or cost?

I know it’s tempting to ignore the performance and just blame “the network” but you own your dependencies and you can do better. Understand your tools, and embrace data structures as persistent storage. After all, in the new serverless world, that’s all you’ve got.


Understanding Block Blobs, Append Blobs, and Page Blobs | Microsoft Docs

 

Categories
development programming

Working exactly to spec

Is there a problem?

  • The work was completed to spec.
  • Any additional work was likely to break the time and cost estimates.
  • The work meets the current standards. To bring the remaining paintwork up to standard would require removing and re-implementing the solution, further risking budgets and blocking higher priority work.
  • The only available colours do not provide the legacy appearance required to match the existing lines.
  • The blue lines were not part of the original specification and therefore no knowledge is available on their purpose and whether they should be extended or removed.
  • The disjointed yellow line on the right-hand side would require straightening, which would cause confusion to existing users. There are multiple consistent configurations and the workers have no means to evaluate which of these minimises confusion.
  • The user who raised the bug report is unaware of the timetable detailing the repainting plan and the agreed extent of the lines.
  • The user who raised the bug report is unaware of future proposed fixes that will require additional upheaval. Any attempt to fix other line issues now will result in unnecessary rework.
  • The existing pattern does not cover the required scope (see the far side), and any additional work would lead to scope creep to correct this error.
Categories
code development programming

Abstractions are scaffolding

All software is an abstraction. It’s human-like language with a logical structure to abstract over ill defined business processes, and gates and transistors, and assembly language, and often 7 network layers, and memory management and databases and a myriad of other things that abstractions paper over to help you deliver business value.

Abstractions are the scaffolding you need to get your project running, but they’re another dependency you need to own. You need to understanding what’s under the abstraction. Or you start thinking network traffic is free. The network is not your friend, it’s slow and unreliable.

Abstractions provide some structure to make what they support easier, but they in turn rely on structures underneath. If you’re going to own an abstraction like any other dependency, you need to understand the structure it’s built on and what it’s designed to support. I understand what ORMs do because I’ve written some of that code in a less performant, less reliable way, before I realised someone had done it a lot better. Indeed, that was the realisation that drove me to alt.Net and NHibernate, but it meant that I understood there was SQL underneath, and the SELECT N+1 problem was a mismatch in the queries that could be resolved, and not just an unexplainable performance hit caused by a network spike.

Abstractions make more sense if you understand what they’re abstracting over. They’re not there for you to forget about what’s underneath, just to save you having to write that code for every class in every project you work on, and to benefit from a wider pool of developers working on battle-tested code. If you don’t know TCP from HTTP, don’t write web applications. If you don’t understand databases or SQL, don’t use an ORM.

All abstractions are imperfect. Learn what’s under the cracks.

Categories
development leadership lifehacks programming

The manager and maker schedule

All technical leads should write code. If you’re not writing code, you’re a manager. And that’s fine. But stop pretending you’re technical. Many Technical Leads or other senior technical staff have a preferred time split between the admin and technical parts of their job, but I don’t always see that in practice.

Some technical leads double down on the technical part and ignore the leadership part. Those are the people who need mentored. It’s the hardest shift when becoming the leader, to accept that your most valuable time is no longer the conversation you have with the compiler, it’s the conversations you have with and on behalf of your team.

You need the time to have those conversations, to keep the team delivering, and delivering the right thing. But you also need up to date technical knowledge to give you the right context for those conversations.

Managing time can be one of the trickiest skills for a new technical lead to learn. The process I use is simple, but it requires discipline.

If you want a split of 60% coding to 40% admin, then schedule it in advance. Pick 2 admin days and 3 development days. Or if you’re most creative in the mornings, schedule 3 hours at the end of each day. Into the admin days put your 1-2-1 meetings and other line management, and your cross-functional meetings, and the team planning and retrospective meetings. Lay out time for emails and backlog pruning and catching up on your reading list. And then block the remainder off for development, deferring to your task board for detail. And make those events as public as you can so your team and others know when they can schedule your time.

Maybe those times are dictated by others and you end up with regular meetings scheduled in your Zone time.If you can’t move it, work the schedule around it. Turn those into your admin days. The most important thing is to define a split, and stick to it.