CodeCraftConf 2019 : What is data anyway? (Answers)

Here’s my thoughts on data following my CodeCraftConf guided conversation. Here are the questions I asked during my guided conversation at CodeCraftConf 2019. They are also available on GitHub if you would like to fork and modify them for your own use.

Most developers are data driven, start with the data structure, not the algorithm. Either data driven design, or the Merise Methodology.

Data, whilst often divided by microservice, is often stored on the same server/cluster, creating a monolith behind the microservices.

Not all data access is secured and audited, although there does appear to be a trend to on-behalf-of flows through the microservice, allowing user-centered access control. Strict data access design is prevalent, although the efficacy was less clear, and strict design applies to all data, including publicly available data.

Keeping sight of data in distributed systems is hard. Jepsen was suggested as one resource to help, but I’m happy to hear of others.

As well as data that can be used to discriminate by collecting gender, name, postcode etc., we also discussed how missing data can be used to discriminate, such as when Glasgow accents aren’t included in voice training data, or when women aren’t used in medical trails.

There’s also the big and growing problem of data collected by people who do not consider the discrimination or privacy implications. For a biologist, DNA is a puzzle that helps them decode cancer, and more examples make the puzzle easier to solve. But for others, DNA is a tool to map insurance risk, to find criminals, and to track down family members whether or not they want to be found. How do we train everyone else to understand?

And the takeaway question : what questions aren’t you asking about your data?

Advertisement

CodeCraftConf 2019 : What is data anyway? (Questions)

Here are the questions I asked during my guided conversation at CodeCraftConf 2019. They are also available on GitHub if you would like to fork and modify them for your own use. Thankyou to everyone who came to the discussion, I will post a follow-up to discuss some of the interesting answers.

What is data anyway?

Navigating SQL, NoSQL, JSON and how to work with data in a post-RDMS, big-data world

Questions

Data modelling

1. When designing a system, do you start with the data or the code?
2. Has the rise of cloud based or non relational data stores changed how we model our data?
3. Do you need to update your data when the models in the code change? How do you do it?
4. Does all your data have to have the same shape?
5. Should the data you expose to the outside world broadly match the data at rest?

Data security

1. How do you secure your data?
2. In light of GDPR, How do you ensure you aren’t collecting too much data?
3. Who has access to your data?

• Do you know if anyone unauthorised has accessed it?

1. How do you protect yourself against bad data and trojan data?

• Bad data = data that is fake, or is used for real world attacks
• Trojan data = data that can compromise your or your customer’s systems

Ethical data

1. Can your data be used to discriminate?

• Can you prove it?
• Is your data biased?
• Are you recording hidden correlations? (ZIP code suggests race)

1. Who owns your data?
2. What questions aren’t you asking?

Unused questions

1. What makes data big?
2. Are you collecting the right data?
3. Is the data you’re collecting right?
4. Where is your data?

Technology choices

1. Do you still have a place for traditional RDBMS?

CodeCraftConf follow-up : notes on growing a team

Whilst I have a few thoughts I’m still processing after the whirlwind of fantastic insights I got from CodeCraftConf, I wanted to capture some of the highlights from some of the answers to my questions on Growing a Team.

Many thanks to everyone who came to the session. Lots of thoughts coming out of the conference.

When is it time to grow your team?

• You never grow a team, you create a new one.
• There’s always too much work, you should grow when you have capacity, so you don’t put new starts off

How do you deal with resistance from existing team members?

• Communicate clearly and address concerns (e.g, time to mentor)
• Involve team early
• Do you have a choice who joins?

Is it more important to find a culture fit or build a diverse team?

• Do candidates know what they’re signing up for?
• Introvert vs extrovert (see also hiring practice)
• Interviews should be structured to filter out arseholes – do they have empathy (or have they been taught to suppress it)
• Every new hire should bring the level up
• Don’t just hire for technical skill
• Diversify your interview pool

What is your biggest worry with your current team size, or with growing your current team?

• Are you doing Health Checks? Survey staff regularly
• Make sure the bigger team, outside your daily work, understand the culture – it only feeds from top down
• Fear of churn
• Loss of culture
• Try to understand
• Make the culture explicit

What practices do you use to ensure sustainable growth?

• Pair a lot, reflect (e.g. retro)
• face-to-face regularly, even if it’s video
• Values workshop – does everyone share them?
• Social convenor to ensure cohesion
• Slack channel dedicated to positive feedback on living the values
• Office/company changes should have their own backlog with an open grooming process so priorities are explicit

How long does it take to integrate someone new?

• Be careful about language
• Train everyone in personal skills
• Listen even when you disagree
• lack of ego
• Culture changes people
• What personalities do you want?

Power corrupts

CodeCraftConf 2018 : Successfully Growing A Team

Thanks to everyone who came to my CodeCraftConf session today, and to the organisers for all their hard work. Here’s the questions I asked, and I’ll follow up with my thoughts from the discussion.

Successfully Growing A Team

1. When is it time to grow your team?
2. How do you deal with resistance from existing team members?
3. Is it more important to find a culture fit or build a diverse team?
4. What is your biggest worry with your current team size, or with growing your current team?
5. How frequently can you add new people to your team?
6. How long does it take to integrate someone new?
7. What practices do you use to ensure sustainable growth?
8. How do you know when a team is too big?
9. How do you split a team that’s grown too big?
10. How do you grow a team when the existing members are already overworked?
11. How long is your recruitment pipeline in terms of short-term planning (getting people in the door) and long-term planning (having the right team in place for next year or 5 years?
12. How do you recruit outside your specialty?

Usable APIs @CodeCraftConf

Thanks for those of you who came to the APIs conversation. If you want to continue the discussion, these are the questions I had on my cards. 

1. What’s the worst thing an API has done to you?
2. What’s the first thing you check when evaluating the use of a new API?
3. What’s the first thing you do when developing a new API?
4. Do you approach developing APIs internally differently to one for the public?
5. How do you make an API foolproof?
6. How much structure should an API have? Does it need a contract? (see SOAP vs REST)
7. How should you change an API?
8. How should you retire an API?
9. Which stakeholders need input to API upgrades and changes, and how do you engage them?
10. How do you keep an API consistent if you grow it via tests?
11. Do bug fixes necessitate a new version?
12. How do you isolate your application from an insecure API?
13. How do you isolate your application from an unreliable API?

CodeCraftConf : call for guides

The CodeCraftConf call for guides is open, and I’d recommend either applying to be a guide or buying a ticket to attend. Last year’s CodeCraftConf was a great event, and I learned a lot.

For those of you unfamiliar with the guided conversations format, the conference website has a good write-up, but it’s a way to keep the conversation going by injecting new questions and riffing of old ones, to give everyone a chance to contribute.

If you want to be a good guide, be sure to share the conversation, and not let one or two individuals control it, especially if they’re in conflict.

If you want to apply, and you’re not sure what sort of questions to ask, have a look at my list from last year, and my write-up of the answers.

I hope to see you there.

More about code reviews

Many thanks to @peitor for engaging with the last code review post, particularly his comments there, which I recommend reading, and his tweet:

@craignicol @yochum Additional to what CodeReviews are for: team building and finding alternative solutions.

— Peter Gfader (@peitor) October 1, 2015

I want to explore a few of his thoughts a bit further, particularly around what code reviews are for.

“team building”

You need to build your reviews carefully to allow team building. As a technical lead, I always get a member of the team, or a peer (if there’s no team yet), to review my code. This can be intimidating for people who believe that there is a hierarchy of knowledge in the team, where the lead knows all, and imparts knowledge. How can such an Oracle be challenged? If I can’t be challenged, I’ve built the wrong team. No-one should release code without review, especially the technical lead, who’s likely to be doing the least coding.

In a properly functioning team however, delegation means that the lead doesn’t have to know everything. They can rely on the developer doing the work to understand the process better than anyone else on the team, and use the review as a learning process to share that knowledge. A functioning code review process not only promotes quality, but it promotes communications within the team.

“finding alternative solutions.”

This is a great use of the review-as-pairing model where the code review is started before the code is complete, allowing for a discussion of options, and the wider context, to ensure the most suitable solution is found.

Maybe you just think of this as a chat between developers, but it’s a great time to review the code and the ideas that generate that code, whilst they’re easier to change. Much easier to change a thought than a test suite.

“What tool can we leverage to make the review more automated?”

I would argue, as @michaelbolton so eloquently does when discussing automated testing with John Sonmez, that the things worth reviewing are the things you can’t automate. Click through and see his replies and the blogs he links to. It’s a gentle but powerful argument.

@michaelbolton Uh oh, someone on the internet thinks I'm wrong. 😉

— John Sonmez (@simpleprogrammr) February 7, 2015

Tools are great. I love compilers, static code checkers, I love the Roslyn examples I’ve seen, but all that comes before the code review. If it doesn’t compile, or it doesn’t meet the style guide, or the tests don’t pass, it’s not ready.

That’s not to say it can’t be reviewed. There may be questions that need reviewed and answered before all the automated stuff passes, but the sign off review requires that the change has passed the automated steps before it can be reviewed.

Also, be wary of following style guidelines. There’s a reason compilers don’t complain about these things. Unless you know why a guideline exists, don’t follow it blindly. Review your automation as much as your code.

“Review not only code in your team. What about the build process? Deployment scripts? Configuration setup?”

Definitely. This might need to involve the whole team, but everything should be open to review and reflect, and where possible, version it so you can review, share and rollback changes. Don’t trust change, but understand it and use it to help you improve.

“Does it stand on its own?”

“What do you mean by stands on its own?As in, the code under review is a complete new feature, or that it is self-consistent (code and tests match, etc.)

Does the change need a lot of explanation via voice? Or is everything there, so that a future reader can follow everything aka the Why? What?”

Does the code leave behind enough context? We all know the scenario where we’re trying to track down some obscure bug, and then we see that 18 months ago, someone used a > rather than a >= and you’re not sure why. The code review is a good chance to document those decisions. If you want to make sure everyone knows you meant “tomorrow or after” instead of “today or after”, make sure it’s explicit, so that when someone calls your code in 12 months time, they don’t get surprised.

Is there anything else I’ve missed?

I love reading your comments, so please let me know

Code Reviews

In my thoughts about codecraft post, I mentioned that we’d discussed Code Reviews during the lean coffee session:

The question was “How much should we nitpick in Code Reviews”, and I can easily see Code Reviews being a great guided conversation on its own, especially with the side discussions on what code reviews are for, who should do them, and when they should be done, which reflect conversations I’ve been having at work.

It’s a topic that’s stuck with me, so I wanted to add a few more questions, to see what you think, and some thoughts about code reviews from my experience.

What are they for?

The common refrain around code reviews is that they ensure consistency and quality of the code, and should be seniors or peers checking the code against a checklist (possibly implied), following a process to ensure that the mistakes that came before aren’t made again. I can hear the agile proponents in the audience screaming at that.

Alternatively, the code review is a structured communication about the problem, and the solution, and the architecture, where the developer feeds back their understanding of the requirement and their implementation of it to another member of the team to validate their understanding, and through that conversation, with another pair of eyes, the developer and their pair can uncover any potential issues that need to be addressed. At the XP end of this scale, this code review is a continuous progress and a natural side effect of paired programming, and in such a view, any review that requires less time than the original development is a poor substitute for a full conversation.

Code reviews can also be used to gate check-ins – to ensure that all code has been viewed by at least 2 (or sometimes 3) pairs of eyes before it is allowed into a branch on the release path.

How much detail should they go into?

If a code review is a conversation, the answer is easy : as much detail as the developer went into, if not more. In the checklist view, the level of detail depends completely on the checklist. Hopefully, the checklist itself is enlightened enough to transfer some of the checking to automated tools, to enforce coding standards, test coverage, and other things, and the code review is the recognition that those automated tools are never enough, and an attempt to plug those gaps.

How long should they take?

As long as they need. It should never be a burden or a blocker to the rest of the team. If code reviews are a bottleneck, there’s something wrong, but equally, a code review that only takes a couple of minutes is a paper exercise, on anything but the simplest change.

Should the developer be present?

The situation is obviously complicated by geographically distributed teams, but let’s assume for this argument, there’s a decent video conference set up for the conversation. The developer has the opportunity to introduce the requirement and the implementation, and the reviewer can ask questions to clarify decisions. Ideally, these would be open questions, that elicit deeper responses from the developer, and may often be generic questions, such as “How does this work?” or “Why did you do choose to to it that way?” that could, if required, be put on flash cards to help new reviewers.

Tools like Github (and gitlab) are set up to allow asynchronous communication over a code review, which lessens the need to a developer to be present, but may depersonalize the experience to some extent, which may provide a more complete check, because the review is of the code rather than the developer, at the expense of potentially alienating developers who don’t agree with the changes. If the developers can’t work together, they can’t review.

If reviews frequently lead to rework, direct conversations are definitely more important to resolve ambiguities and ensure the developer being reviewed understands the reasons and consequences.

Should they be done at the end or the start of the feature?

This was a way of working that I discovered from Gitlab, where many pull requests are created at the same time as the feature branch, allowing conversations between developers as the code is developed, rather than using code reviews to gate check-ins prior to merging. This allows architectural and other design decisions, and the requirements, to be discussed in situ, with the code as it is being developed, which should inoculate the branch from the worst effects of drift that may occur once the Continuous Integration ideal of everyone integrating onto the same branch multiple times a day is compromised to allow feature branching.

Thoughts about @codecraftuk conference

I really enjoyed the CodeCraftConf last Friday. Well done to the organisers for a great set of discussions. A very different style of conference, and a refreshing change from the lecture-based conferences. A lot of good discussions, and I just wanted to throw some thoughts down whilst it’s still fresh in my head.

technical lead discussion

Technical lead discussion summary available here.

agile

The questions in this session were based around the Agile Manifesto:

Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan

That is, while there is value in the items on
the right, we value the items on the left more.

There was a lot of folk agreeing with the things on the left, but it was hard to find people who saw much value in the things on the right until directly challenged on it, which I found an interesting dynamic. There was also a lot of talk about feedback and communication and how that works. Plenty of interesting thoughts.

lean coffee

Prior to lunch, I joined the lean coffee session, where a number of topics were proposed.

There were a few topics covering mobile development, javascript frameworks and some other good topics, but the one that sticks out most for me was about Code Reviews.

The question was “How much should we nitpick in Code Reviews”, and I can easily see Code Reviews being a great guided conversation on its own, especially with the side discussions on what code reviews are for, who should do them, and when they should be done, which reflect conversations I’ve been having at work.

I think it’s a great format, and I look forward to trying it at our next lunchtime session in the office.

security

This session particularly interested me as I used this topic as a dry run in the office before I guided my session on the day, and regular readers will notice that security is an interest of mine. The key questions here were around application security, but a lot of the discussion covered testing and social engineering, and why we still trust some companies that have been hacked, but not others.

BDD

I had trouble deciding which session to attend at the end of the day, but as I was late in, the decision was made for me, as there were no seats left at the other talk. So there was a slight musical chairs element to proceedings.

The session itself was particularly interesting to me as I’ve been exposed to some of the ideas and technologies related to Behaviour Driven Development, but I haven’t had a chance to discuss using it for actual projects, and following the discussion, I understand that it requires a particular relationship between the developers and product owners to work. Definitely something to explore further.

There was another interesting discussion in this session, that had been bubbling through most of the sessions in the day – the idea of “Cargo Cult process”, where the memes and practices of techniques such as TDD, Agile and BDD have been parachuted in to a project, or organisation, without understanding the purpose of the techniques or the problems they are trying to solve. As discussed in the Agile session, the successful processes are those that start with a problem, a pain point, and find ways to address that, and put a feedback loop in to ensure the problem is addressed, and then move on to the next pain point.

If customer engagement isn’t your problem, the BDD Three Amigos session might not give you much value. If your product owner trusts your developers, or can read CI results, you may not need Cucumber to write your tests.

Venue

I was a little but unsure about a conference in a pub, but there was a great space, and some of the best conference food I’ve had. The room itself could get a little noisy when the conversations got heated, as there were 2 converations going on at once, but the organisers were working to address that on the day and at future

I’m looking forward to the next one.

Technical Leadership at @codecraftuk follow up

I had a great time at the CodeCraftConf, and I’ll be writing up my thoughts on the other sessions shortly, but for the benefit of those who attended the technical leadership session and those who were interested in the questions, I made a few notes on each question to reflect suggestions, and discussion points that we covered, as well as a couple of additional questions that came up during the conversation. Feel free to use them as inspiration for running your own session around these questions.

Before that however, I just wanted to give a few thoughts on guiding a session, for anyone interested in running one themselves.

Preparation

The best questions are open, and not technology specific. I also found that phrasing similar questions in different ways can help to draw out additional detail. I had the questions reviewed by a couple of trusted friends in advance, to make sure they covered enough ground and were good starting points for discussion.

Guiding

I chose the first question as an ice breaker, as it was something I felt anyone interested in the topic could give a short answer too. Don’t be precious about the order of questions. Know what’s coming up so you can pull questions forward when the conversation naturally flows into it.

Try and pick up the subtle cues for people who want to speak, and use your authority as guide to try and spot them. This may often mean liking away from who’s currently speaking.

---

What is a technical lead?

Role model
Learning
Build your relationships
Technical Architecture
Decision maker
How to build software better
Facilitate not dictate
Coach / mentor
Coding standards
Person I go to when I have a problem
Encourage communication and collaboration
Challenge and grow the team
Arbitration and negotiation
Scrapper – fighting external stakeholders in behalf of the team.

What are the most important behaviours for technical leads to exhibit?

Calm under pressure
Promoting knowledge
Approachable
Don’t have to know everything. Knowledge is an inverse hierarchy, it collects at the bottom.
Balance
Drawing out quieter team members
Delegation
Open minded

What most inspired you about your previous technical leads?

Clarity on code knowledge
Removal of fear
Encourage learning from mistakes – mistakes happen, so learn from them when they do.
Protect from blame – collective ownership, no scapegoats
Guiding rather than fixing – not grabbing control of the keyboard
Tailoring messages to the audience
Simplifying solutions.

Why do you want to be a technical lead?

“I didn’t want to be a technical lead ”
Help people
Recognition
Drift into it
Job description makes it uncomfortable / or gives you ammunition when scrapping
Earned authority

What scares you most about being a technical lead?

Responsibility – easy to make big mistakes
Not keeping up with technology
Friction with the outside and non-technical stakeholders
Being reactive vs proactive
Meetings
Deskilling

How do you measure success as a technical lead?

If nobody needs you
Happy management – no meetings
Happy team

What one thing would make your life as a technical lead easier?

Open minded developers
Customers who know what they want and what the requirements are
Unified working practices and strong management
Ban the word “just”
100% utilisation is bad
More time to just think
The right work environment
Flexibility and respect
The right tools

How much coding should a technical lead do?

A lot!!
Structured time boxes
Lead on one project, developer on the next
1/3rd – 2/3rds coding was the accepted range in the session, but some unhappy leads were doing less
“feature lead”

What responsibilities are you happy to delegate, and what do you want to control?

Delegate everything. But know it will be OK.
Control to make sure the delegated work is complete and the team feel supported.
Map delegations to the team
Delegate by negation – military strategy :
– “I will do this”, give leader option to refuse, then carry on
– This style can take a while to get used to
Constant communication
Don’t micromanage

How do you plan for your own absence, so you can rest on holiday?

Good team mix – senior vs junior
Choosing your staff
Building the right relationships internally and externally
Documentation
Automation

What qualities do you want your team to have, and how to you help them get that?

Ran out of time to cover this question, skipped to a combination of the following two.

How do you deal with conflicts in the team?
How do you deal with external pressures on the team?

Life experience
Soft skills
Project Manager should be the political front to the outside world
Have to work in partnership with the PM
Compassionate, put yourself in their shoes
Smaller teams are easier

Additional questions :

What happens to the project when the technical lead changes?

Do you need a tech lead? Are techniques like voting to resolve conflicts enough?