Categories
code development programming security

This is raw

This is raw chicken : 🐤

If you eat it like that, you may get hurt immediately, by its beak, or its claws. It may grab your money and run off with it.

If you want to eat it, better to kill it first. 💀

If you eat it like that, you may get hurt or die, in a few hours, or days. Washing it won’t help.

Cook it. Cook it well. If there’s any sign of pink, cook it some more. 🔥

It might still kill you, but at least you’re a lot safer than when you started.


This is raw data : 🐤

If you display it, users will get hurt immediately, whether by cross-site scripting, cookie sniffing, crypt-currency mining, or something else. If you’re lucky, it will be something your user’s see immediately and leave your site never to return. Otherwise they may get infected.

If you want to use it, better to validate it first. 💀

If you save it like that, your users are still vulnerable. It might appear on the front end in a different form. It might be a string of unicode characters that crashes your phone. It might be a link to somewhere they can’t trust.

Encapsulate it. Sandbox it. Never trust it, in or out . HTML encode, whitelist the output as well as the input.

And if you need to avoid spam, or incitement, or solicitation, maybe you need editors. Computers can’t fix all the social problems. 🔥

Categories
code development programming

Translations with React, Redux and Asp.Net

In a recent project, which was the first time I’ve used React and Redux in anger, we had a requirement to support 2 different languages, in the .Net backend for emails and PDFs, and in the React frontend.

As the translators we used were used to resx files, we wanted to use those as the master source. In other projects we’ve done the javascript translations via a pre-compilation step into static javascript files, but since redux has it’s own store, I decided to see if we could use that to store and process the translations.

This approach has the advantage that we can use the redux state to translate the site automatically when a new language is selected, without having to reload any pages. For an application that depends on up-to-date data, and had to operate on por data connections, avoiding reloads is essential.

The redux examples shown here use TypeScript, which certainly helped in our development process, but there’s a lot of gotchas getting the dotnet new react-redux template and many 3rd party react and redux libraries working nicely with TypeScript. You can make it work, but it’s definitely not an out-of-the-box solution in most cases.

The solution consists of 3 parts:

  1. The Asp.Net Core 2.0 controller to translate the resx data into JSON (in this example, the resource file is called UserFacingStrings.resx and the default is en-US,
  2. The Redux configuration to retrieve the data and populate it into the store; and
  3. A React function that reads the translation from the store and presents it to the user.

The solution below attempts to get the localisation from the user headers, so you’ll need to enable localisation in your Startup.cs, but depending on your use case, you may want save user language settings in the user’s account data, or in a cookie or other storage in their browser.

ASP.Net Core API controller

[Produces("application/json")]
[Route("api/Localisation")]
public class LocalisationController : Controller
{
    private static Dictionary _cultureCache = new Dictionary();

    /// Get all known translations
    /// Choose a culture (e.g. `en-US` )
    /// will default to browser culture if not specified
    [HttpGet("[action]/{culture?}")]
    public IActionResult Translations(string culture)
    {
        try
        {
            var cultureInfo = CultureInfo.CurrentUICulture;
            try
            {
                string preferredLanguage = HttpContext.GetPreferredLanguage();
                if (!string.IsNullOrWhiteSpace(culture))
                {
                    cultureInfo = CultureInfo.GetCultureInfo(culture);
                }
                else if (!string.IsNullOrWhiteSpace(preferredLanguage))
                {
                    cultureInfo = CultureInfo.GetCultureInfo(preferredLanguage);
                }
            }
            catch (CultureNotFoundException)
            {
                // Fallback to en-US
            }

            if (_cultureCache.Keys.Contains(cultureInfo.ToString()))
            {
                return new OkObjectResult(new { t = _cultureCache[cultureInfo.ToString()] });
            }

            var translations = new Dictionary();

            // Insert all resources, to then be overwritten
            if (cultureInfo.TwoLetterISOLanguageName != "en")
            {
                ExtractResources(translations, UserFacingStrings.ResourceManager.GetResourceSet(CultureInfo.GetCultureInfo("en-US"), true, true));
            }

            ExtractResources(translations, UserFacingStrings.ResourceManager.GetResourceSet(cultureInfo, true, true));

            _cultureCache[cultureInfo.ToString()] = translations;

            return new OkObjectResult(new { t = translations });
    }
    catch (Exception)
    {
        return NotFound();
    }
}

private static void ExtractResources(Dictionary translations, ResourceSet resourceSet)
{
    foreach (DictionaryEntry res in resourceSet)
    {
        translations[res.Key.ToString()] = res.Value.ToString();
    }
}

Browser Extension

public static class BrowserDetailsExtensions
{
    public static string GetPreferredLanguage(this HttpContext http)
    {
        return http?.Request
            ?.Headers["Accept-Language"].ToString()
            ?.Split(',').FirstOrDefault()
            ?.Split(';').FirstOrDefault();
    }
}

Redux plumbing

CultureActions.ts

import { AppThunkAction } from '../store/index';
import { fetch } from 'domain-task';
import { CALL_API } from 'redux-api-middleware';
import { actionCreatorFactory } from 'typescript-fsa';
import { Translations } from '../reducers/CultureState';
import { Dictionary } from 'lodash';

const API_ADDRESS: string = "/api/Localisation/";
const API_GET_TRANSLATIONS: string = "Translations/";
const API_LANGUAGE_CODE: string = ""; // "pt-BR", "en-GB"
const API_REQUEST_TYPE_GET: string = "GET";

export interface ApiError {
    name: 'ApiError',
    status: number,
    statusText: string,
    response: string
}

export interface GetTranslationsAction {
    type: 'RECEIVE_TRANSLATIONS';
    t: Dictionary;
};

export interface TranslationsApiPosted {
    type: 'Translations_REQUEST_POSTED';
    payload: never;
};

export interface TranslationsApiFailed {
    type: 'Translations_FAILURE';
    payload: ApiError;
    error: true;
};

export interface TranslationsApiSuccess {
    type: 'Translations_SUCCESS';
    t: Dictionary;
}

// Declare a 'discriminated union' type. This guarantees that all references to 'type' properties contain one of the
// declared type strings (and not any other arbitrary string).
export type CultureAction = GetTranslationsAction;

const actionCreator = actionCreatorFactory();
export const translationsAwaitingResponse = actionCreator('Translations_REQUEST_POSTED');
export const translationsSuccess = actionCreator('Translations_SUCCESS');
export const translationsFailure = actionCreator('Translations_FAILURE');

// ACTION CREATORS don't directly mutate state, but they can have external side-effects (such as loading data).
export const cultureActions = {
    getTranslations: (): AppThunkAction => (dispatch, getState) => {
        console.log("REQUEST TRANSLATIONS ACTION")
        const rsaaRequestAuctions = {
            [CALL_API]: {
                credentials: 'same-origin',
                endpoint: API_ADDRESS + API_GET_TRANSLATIONS + API_LANGUAGE_CODE,
                method: API_REQUEST_TYPE_GET,
                types: ['Translations_REQUEST_POSTED', 'Translations_SUCCESS', 'Translations_FAILURE']
            }
        }
        dispatch(rsaaRequestAuctions);
    },
};

CultureStore.ts and the tr() method

import { Dictionary } from "lodash";
import * as He from 'he';
import { Decimal } from "decimal.js";

export type CultureState = {
    strings: Translations
};

export interface Translations {
    t: Dictionary
};

export const emptyCulture: CultureState = {
    strings: {
        t: {}
    }
};

export function tr(culture: CultureState, key: string): string {
    try {
        return He.decode(culture.strings.t[key]) || key;
    } catch (error) {
        console.log("translation for '" + key + "' not found");
        if (key == undefined || key == null) {
            return "- -";
        }
        return "-".repeat(key.length);
    }
}

// Returns the key value itself if no match is found in the resx
export function trFallback(culture: CultureState, key: string): string {
    try {
        return He.decode(culture.strings.t[key]) || key;
    } catch (error) {
        return key;
    }
}

export function trFormat(culture: CultureState, key: string, args: string[]): string {
    return formatString(tr(culture, key), args);
}

export function formatDecimalAmount(subject: Decimal): string {
    return formatAmount(subject.toNumber());
};

export function formatAmount(subject: number): string {
    return subject.toLocaleString(navigator.language, { maximumFractionDigits: 2 });
};

function formatString(subject: string, args: string[]): string {
    if (subject === undefined) {
        return "";
    }
    return subject.replace(/{(\d+)}/g, function (match, number) {
        return typeof args[number] != 'undefined'
            ? args[number]
            : match
        ;
     });
};

CultureReducer.ts

import { Action, Reducer } from 'redux';
import { isType } from 'typescript-fsa';
import { Translations, emptyCulture, CultureState } from './CultureState';
import { ApiError, CultureAction, translationsAwaitingResponse, translationsFailure, translationsSuccess } from '../actions/CultureActions';
import { Dictionary } from 'lodash';

export interface CultureStateR extends CultureState { }

const unloadedState: CultureStateR = emptyCulture;

export const reducer: Reducer = (state: CultureStateR, incomingAction: Action) => {
    const action = incomingAction as CultureAction;

    if (typeof state === undefined) {
        return unloadedState;
    }

    if (isType(incomingAction, translationsAwaitingResponse)) {
        return {
            strings: state.strings
        };
    }
    if (isType(incomingAction, translationsFailure)) {
        return {
            strings: state.strings
        };
    }
    if (isType(incomingAction, translationsSuccess)) {
        const rawTranslations: Dictionary = incomingAction.payload.t;
        return {
            strings: { t: rawTranslations }
        };
    }

    switch (action.type) {
        case 'RECEIVE_TRANSLATIONS':
            return state;

        default:
            break;
    }

    return state || unloadedState;
}

Use in account controller

import { CultureState, tr } from '../reducers/CultureState';
/// Some imports removed for clarity

/// Add CultureState into props for this component
type AccountProps = AccountState & CultureState & typeof accountActions & RouteComponentProps;

export class Account extends React.Component {
}

Use tr() in view tsx file

</pre>
<h2>{tr(this.props, "PersonalDetails")}</h2>
<table>
<tbody>
<tr>
<td>{tr(this.props, "Name")}:</td>
<td>{this.props.user.userName}</td>
<td><a href="/account/profile">{tr(this.props, "Edit")}</a></td>
</tr>
<tr>
<td>{tr(this.props, "Company")}:</td>
<td>{this.props.user.companyName}</td>
</tr>
<tr>
<td>{tr(this.props, "Email")}:</td>
<td>{this.props.user.emailAddress}</td>
</tr>
</tbody>
</table>
<pre>
Categories
.net development

Debugging Asp.Net Core 2 apps in Azure

I’ve been getting under the skin of Asp.Net Core 2 following a failed experiment with Asp.Net Core v1. I certainly found .Net standard and the associated framework support to be very welcome, and unlike my previous project we didn’t need to support GDI or SignalR libraries, so it’s much more in the Core sweet spot.

I’ll talk about the project and the technologies involved in some follow up posts, so if you’ve got any real-world questions on Asp.Net core, React/Redux with Typescript, or CosmosDb in C#. let me know and I’ll try and address them as I get to those technologies.

For now, though, I want to start with the basics. Asp.Net core on Azure works great, mostly, but if something goes wrong in Startup.cs, there’s no Application Insights, a generic 502.5 IIS error – which just means it can’t talk to Kestrel, and no web logs to help you. So before you deploy to Azure, do yourself a favour and add the following to web.config so you’ve got logs to help you.

<system.webServer>
<handlers>
<add name="aspNetCore" path="*" verb="*" modules="AspNetCoreModule" resourceType="Unspecified" />
</handlers>
<aspNetCore processPath="dotnet" arguments=".\web.api.dll" stdoutLogEnabled="true" stdoutLogFile=".\logs\stdout" forwardWindowsAuthToken="true" />
</system.webServer>

That way, if you do see a 502.5 error on your site, you can jump into Kudu and start reading the logs. They can grow quite quickly, depending on your web app lifecycle settings, so you may benefit from a regular cleanout of your logs folder.

If the logs still aren’t helping, or you don’t understand what you’re seeing, there’s a nice Asp.Net core 2 troubleshooting guide over at MSDN, but not all of it applies to Azure.

Categories
development programming security

NMandelbrot : running arbitrary code on client

As part of my grand plan for map-reduce in JavaScript and zero-install distributed computing, I had to think about how to gain user trust in a security context where we don’t trust the server. I couldn’t come up with a good answer.

Since then, we’ve seen stories of malicious JavaScript installed to mine cryptocurrencies , we know that JavaScript can be exploited to read kernel memory, including passwords, on the client, and I suspect we’ll see a lot more restrictions on what JavaScript is allowed to do – although as the Spectre exploit is fundamentally an array read, it’s going to be a complex fix at multiple levels.

I had ideas about how to sandbox the client JavaScript (I was looking at Python’s virtualenv and Docker containers to isolate code, as well as locking them into service workers which already have a vastly more limited API), but that relies on the browser and OS maintaining separation, and if VMs can’t maintain separation with their levels of isolation, it’s not an easy job for browser developers, or anyone running on their platform.

I also wondered if the clients should be written in a functional language that transpiled to JavaScript, to have language level enforcement of immutability and safety checks. And of course, because a functional style and API provides a simpler context to reason about map-reduce, by avoiding any implicit shared context.

Do you allow someone else’s JavaScript on your site, whether a library, or a tracking script, or random ads from Russia, Korea, botnets and script kiddies? How do you keep your customers safe? And how do you isolate processes you trust from processes that deal with the outside world and users? JavaScript will be more secure in the future, and the research is fascinating (JavaScript Zero: real JavaScript, and zero side-channel attacks) but can you afford to wait?

Meltdown and Spectre shouldn’t change any of this. But now is a good time to think about it. Make 2018 the year you become paranoid about users, 3rd parties and other threats. The year is still young, but the exploits are piling up.

 

Categories
development

New Job : Welcome to Screenmedia

Following my request across my network looking for a new job, I started at Screenmedia 3 weeks ago. For those who don’t know, it’s a digital design practice, which means I’m back to consulting, and I get to work with a lot of smart people, covering technology and design. I’m a Technical Architect in the integrations team, so that’s APIs, voice assistants, serverless, analytics, so should be a good wee adventure. I’ve got a few thoughts on the job hunt which I’m sure will come up at ddd.scot and future blog posts. But if you’re currently looking, we’re hiring. If you’ve got any burning career questions, the DDD Scotland panel survey is still open.

I’ve been working on lots of interesting projects already, so groking multiple domains, sometimes multiple for the same customer, Checking the checklists, and reviewing the onboarding process. Sometimes the best change is the one that lets you re-evaluate what you think you know.

 

 

Categories
development ux

User Experience : A quick introduction for developers

The following is an internal summary I wrote for a team that no longer exists, summarising a number of references from UXScotland, various book and blog posts. It extends the thoughts from my Pecha Kucha talk. For more details, please refer to the links throughout and the references at the bottom. The context here is consulting and long-term B2B projects, but some of these discussions are more widely relevant.

User Experience : Project considerations

User experience is about making sure we are solving the right problems in the right way. It is the intersection of design, users, and context. Context here is a combination of one or more of the device in use, the user’s location, any social cues such as nearby friends, and anything else that may be available from sensors or historical information.

vennux

In many cases, the requirements we have are assumptions (e.g. what users want is Facebook integration). Where the benefits of a requirement are unclear, we should treat it as an assumption to be tested. Embrace data, and analyse it.

At the requirements stage, we need to make sure we are solving the right problem (pretotyping : “building the right *it*”), and that our chosen design helps the user to solve the problem without frustration (i.e. prototyping the design, rather than the implementation, with wireframes/sketches).

In user testing, particularly in an agile development, we can refine those ideas by seeing how well the implementation solves the problem, by testing with users. We can also test deployed code by analysing heat maps and http logs to see what users are doing to inform further tests and the assumptions that feed into further design cycles.

In a Lean/Agile project, we need to be explicit about our assumptions about the user and test them at every stage of the development to ensure that we always meet user needs.

cycleofux

How does UX fit in our process?

Stakeholders

A system that supports user’s need effectively will need to understand that the user is a Stakeholder in the process. Whilst the users themselves may not be directly involved in the generation or review of design artefacts, there should be a user representative, either a super-user on the customer side, or a 3rd party researcher who has determined user needs, and has authority to verify any proposed solution and high-level requirements against those needs.

Functional Requirements

Personas / Typical Users

A persona is an abstraction of a system user. In a simple system, there may be only one type of user, but more sophisticated systems will typically have users and administrators, and may have multiple classes of each. A persona is defined to encapsulate the types of tasks a specific user may wish to perform, and any limitations that may be imposed (for example, administrators may be able to install specific browsers or client software, but members of the public using the system must be supported across multiple browsers at multiple screen sizes).

User Journeys

Each persona will have one or more tasks they wish to perform in the system. A User Journey describes the tasks as a series of steps that a specific persona will follow in order to achieve that task.

Consider the tasks that a user wants to perform. See also BDD – design from user in.

E.g.

User wants to process a case:

  • User logs in to the system
  • User selects case from their task list
  • User reviews latest document
  • User finds agent for case, and calls to discuss
  • User adds comments to case
  • User saves case and returns to their task list

This process may identify a new use case (“Display task list”), and specific actions that need to be defined within a use case (“Display latest document” and “Display agent contact details”)

The User Journeys provide the context between the Stakeholders (and User Types therein) and the Use Cases. Each User Journey will link to one or more Use Cases, but some Use Cases may not have an associated User Journey (nightly payment processing, for example).

User feedback

If the solution is replacing or improving an existing system, the best source of information on the current system are the users. The requirements capture process should take into account both the tasks that the users perform and gather feedback on any areas of frustration. The prioritization exercise should consider these improvements as well as new functionality.

Testing

As well as testing the Use Cases for functional acceptance, the FAT/UAT process should also test that the final system supports the User Journeys defined up front.

On-going support

Where projects have regular support meetings, the input of users has been valuable in identifying problems areas and possible changes. When on-going service delivery contracts are defined, SDMs should consider whether ongoing user feedback is appropriate as part of the planning and scoping of releases within that framework.

Questions to ask

  • Have the requirements been tested on users? If not, why not? (Are these the right requirements?)
  • Will users be given the opportunity to provide feedback on these through the development? (And if so, how, when and where?)
  • What user outcomes are we trying to achieve with the release? These may not be requirements that we put a cost on, but an expectation that we can measure against to show improvement – we would need to communicate this appropriately.
    • E.g. minimise clicks to access the 5 main functions
    • E.g. reduce time-to-complete for function x, y and z by 10%
    • E.g. Align existing UI with iOS and Android norms
    • E.g. Increase usage of function z by 5%
    • E.g. 99% AAA compliance
  • Who represents users on the project team?
    • How many user types do we need?
    • Can normal users and administrators share UX, or are their goals divergent? – different apps, different ASP Areas, different branding, …
  • What platforms and form factors need to be supported/tested?
    • Does each platform need a native UX? If native app, probably yes, if web app, maybe.
    • If mobile, do we need to adapt to context : location/orientation/communication with nearby devices/…
    • If social, do we need to adapt to context : can I approve my own work?/who’s online/recommendations/who’s nearby/…
  • Do we, as developers, have any input to the UI design? If not, why not?
  • Have the designs been tested on users? If not, why not? (Does the UI fit user expectations?)
  • Do we have appropriate guidelines for the appropriate platform, and are they listed in the requirements and estimates?

Potentially useful resources

 

Categories
code development programming

Abstractions are scaffolding

All software is an abstraction. It’s human-like language with a logical structure to abstract over ill defined business processes, and gates and transistors, and assembly language, and often 7 network layers, and memory management and databases and a myriad of other things that abstractions paper over to help you deliver business value.

Abstractions are the scaffolding you need to get your project running, but they’re another dependency you need to own. You need to understanding what’s under the abstraction. Or you start thinking network traffic is free. The network is not your friend, it’s slow and unreliable.

Abstractions provide some structure to make what they support easier, but they in turn rely on structures underneath. If you’re going to own an abstraction like any other dependency, you need to understand the structure it’s built on and what it’s designed to support. I understand what ORMs do because I’ve written some of that code in a less performant, less reliable way, before I realised someone had done it a lot better. Indeed, that was the realisation that drove me to alt.Net and NHibernate, but it meant that I understood there was SQL underneath, and the SELECT N+1 problem was a mismatch in the queries that could be resolved, and not just an unexplainable performance hit caused by a network spike.

Abstractions make more sense if you understand what they’re abstracting over. They’re not there for you to forget about what’s underneath, just to save you having to write that code for every class in every project you work on, and to benefit from a wider pool of developers working on battle-tested code. If you don’t know TCP from HTTP, don’t write web applications. If you don’t understand databases or SQL, don’t use an ORM.

All abstractions are imperfect. Learn what’s under the cracks.

Categories
cloud development

Cloud thinking : efficiency as a requirement

In the old world, you bought as big a machine as you could afford, and threw some code at it. If it could fit in memory and the disk I/O wasn’t a bottleneck, everything was golden.

In the cloud, however, CPU cycles and disk storage cost real money. Optimisation is key, so long as it’s not premature. Monitor it.

In cloud thinking, it’s less about the O(N), it’s relatively easy to scale to the input, as long as you’re not exponential. In the cloud, it’s about O($) – how well does your code scale to the amount of money you throw at your servers (or inversely, what’s the code increase per user).

Fixed costs are vanishingly small in the cloud, but incremental costs can change quickly, depending on your base platform. Not the provider, as costs between them are racing to the bottom, but the platform of your architecture.

Quite simply, the more control you ask for, the more you pay in time, and the bigger the ramp-up steps. Get metal, and you’ll pay for everything you don’t use, get a VM and you can scale quicker to match demand, get a container, get an Electric Beanstalk or an Azure website and give up the OS, get a Lambda or a Function.

I can’t recommend to you how much you should abstract. I don’t know how big your ops team are, or how much computation you need to do for each user. I suspect you might not know either. Stop optimising for things you don’t care about. Optimise for user experience and optimise for cost per user, and measure everything.

Categories
code development

Excellent Export, IE and Security

Following on from supporting large Client-side spreadsheets in Chrome, I’ve extended my pull request to also support IE, which uses its own proprietary Javascript method. Because, Microsoft.

And all was good with the world.

Until a Microsoft update broke HTML-formatted XLS files. Because, Microsoft. If you support XLS downloads on your site, and your users are getting a blank Excel window when they try and open them, thank Microsoft, and a security fix. Then go back to using CSV. Or ODS.

Thanks Microsoft.

Categories
code development

Excellent Export and the Chrome URL limit

One key difference between junior and senior developers is that when they encounter a bug, senior developers are much more likely to blame themselves before others, because experience moves you up the Dunning-Kruger curve.

I was given a bug report recently where Chrome was unable to download a particular spreadsheet, and IE didn’t work either. Other spreadsheets worked OK. It was fairly obvious that one feature of this spreadsheet was the likely candidate : it was by far the largest spreadsheet in the system.

I tried to recreate the bug, and I use Firefox, because I prefer the developer tools. And the spreadsheet download worked as expected. So it’s a difference between browsers. It might still be my fault, but it’s looking less likely.

The library we used for the export is a Javascript library called excellentexport, which converts an HTML table into an Excel-compatible spreadsheet. It uses the data: URI scheme to encode the spreadsheet directly into the page, so that the spreadsheet can be generated entirely server-side. Internet Explorer does not support data: so displays the link, but clicking on it has no effect. Chrome and Firefox both support it, but it turns out Chrome reduced the URI limit from (what I think is) 256Mb to 2Mb, and I wasn’t able to see why. This originally caused the tab to crash, and still can under the right conditions, but for URIs just bigger than 2Mb (in our case 3.5Mb), the click is processed, but the data URI is rendered empty, so a null file is downloaded.

This can be a tricky one to diagnose, hence this post. If you ever encounter this, the solution is to switch to CreateObjectUrl, which is supported in all recent browsers, including Edge, but not IE (but as it doesn’t support data: you’re not losing out there). I’ve submitted a Pull Request to excellentexport demonstrating the change required. Worth remembering if you ever need to create a download link client side.