Notes from 1/7–7/7

Artificial Intelligence

GPT’s downfall was that it was pre-trained using traditional language modeling, i.e., predicting the next word in a sentence. In contrast, BERT was pre-trained using masked language modeling, which is more of a fill-in-the-blanks exercise: guessing missing (“masked”) words given the words that came before and after.

Interesting to see that GPT generates biographies with a bias on the perceived ethnicity and sex associated with a name.

The blog basically used Tensor2Tensor to visualise how GPT-2 made decisions.

BERT makes use of Transformer, an attention mechanism that learns contextual relations between words (or sub-words) in a text.

Masked LM

Before feeding word sequences into BERT, 15% of the words in each sequence are replaced with a [MASK] token. The model then attempts to predict the original value of the masked words, based on the context provided by the other, non-masked, words in the sequence

Next Sentence Prediction

the model receives pairs of sentences as input and learns to predict if the second sentence in the pair is the subsequent sentence in the original document. During training, 50% of the inputs are a pair in which the second sentence is the subsequent sentence in the original document, while in the other 50% a random sentence from the corpus is chosen as the second sentence.

You can also finetuned BERT to some other tasks such as NER, Question answering and classification. Okay but it takes 4 days to train.

Train a classifier model using open 3D model datasets to predict what sort of 3D model is being built, then recommend them to the designer. I suppose a similar concept can be used when it comes to other things that require some sort of prompting.


I wouldn’t exactly call these decision models per se. More of conceptual frameworks. However, they are really high level and requires some unpacking to actually answer the question.

Cyber security

ScrapedIn — to search LinkedIn

Simple tool to exploit people who, in their folly, openly declare they have top secret security clearance in order to get jobs in security companies.


Okay maybe it’s time to do a deep dive into this. why not eh?



man [COMMAND] to view the arguments supported

cat [FILE_PATH] opens up the file

head [FILE_PATH] or tail [FILE_PATH] to see only the top or bottom lines

wc -l / -w /-c [FILE_PATH] that lets you count the lines -l , words -w or characters -c

grep "[SEARCH_STRING]" [FILE_PATH] printing all lines that contain your search string

| piping data from one command to the next

sort -t "[DELIMTER]" -k [COLUMN] -r -n [FILEPATH] sorts contents of the file according to the delimiter, the column to sort on, -r is for descending sort (it sorts by ascending by default) and -n for numerical sorting

uniq used when you pipe data from a sort operation

cut -d "[DELIMITER]" -f [COLUMNS] [FILE_PATH] to view specific columns

tr "[CURRENT]" "[REPLACEMENT]" replace characters

find . -name "A*.csv" this is a search command that lets you find all the files strating with A. * is a wildcard

> [FILE_NAME.txt]write into another file

Great way to explain a typical probability question. All about using estimators. Mentions your Maximum Likelihood Estimate (MLE) and your Minimum Variance Unbiased Estimator (MVUE)


Foucault suggests that there is no fixed, ahistorical notion of human nature, as posited by Chomsky’s concept of innate human faculties. Chomsky argues that concepts of justice are rooted in human nature and reason, whereas Foucault rejects any such universal basis for a concept of justice.

Foucault is really radical in this aspect. Justice, from what I understand, comes from human reason. Only then can we decide what constitutes a just act. Chomsky was quite insistent that a just society must promote the truest aspects of human nature. But, human nature might just be impossible to define. Thus, socity should promote an ideal concept of human nature but still be open to changes. Now that, fundamentally will be very difficult due to the way institutions are created to enforce these idealised notions of human nature itself.

Product Management

We are developing a Facebook Messenger chatbot that helps migrant to:

Connect with a community and social support network in their arrival city.

Maintain, manage, and/or get new documentation or registration upon arrival.

Match employment based on their skills.

Affordable housing information.

Interesting concept. It automates a lot of these painful tasks that are highly consuing for non locals.

Quick dive into SEO.

So that’s what SEO features are for. It’s just a way to optimise your site for Google’s crawlers.


Decision-making framework that can be adapted, something useful to take a look?

  1. Transparency, which empowers employees to make their own decisions
  2. Intellectual Rigor, which helps the company uncover innovative ideas
  3. Radical Candor, which helps us to grow fast

Makes one responsible for their own decisions and rewards initiative. But allows one to receive frequent feedback so refine these said ideas.

Hindsight biases post-accident assessments of human performance. Knowledge of the outcome makes it seem that events leading to the outcome should have appeared more salient to practitioners at the time than was actually the case. This means that ex post facto accident analysis of human performance is inaccurate. The outcome knowledge poisons the ability of after-accident observers to recreate the view of practitioners before the accident of those same factors. It seems that practitioners “should have known” that the factors would “inevitably” lead to an accident.2 Hindsight bias remains the primary obstacle to accident investigation, especially when expert human performance is involved.

Perfect way to analyse incidents. Qualifies the concept that hindsight is 20/20

Actions at the sharp end resolve all ambiguity. Organizations are ambiguous, often intentionally, about the relationship between production targets, efficient use of resources, economy and costs of operations, and acceptable risks of low and high consequence accidents. All ambiguity is resolved by actions of practitioners at the sharp end of the system. After an accident, practitioner actions may be regarded as ‘errors’ or ‘violations’ but these evaluations are heavily biased by hindsight and ignore the other driving forces, especially production pressure.

So… ultimately organisations create an illusion of order. It allows for confidence to act rather than to create the parameters for acting.

Change introduces new forms of failure. The low rate of overt accidents in reliable systems may encourage changes, especially the use of new technology, to decrease the number of low consequence but high frequency failures. These changes maybe actually create opportunities for new, low frequency but high consequence failures. When new technologies are used to eliminate well understood system failures or to gain high precision performance they often introduce new pathways to large scale, catastrophic failures. Not uncommonly, these new, rare catastrophes have even greater impact than those eliminated by the new technology. These new forms of failure are difficult to see before the fact; attention is paid mostly to the putative beneficial characteristics of the changes. Because these new, high consequence accidents occur at a low rate, multiple system changes may occur before an accident, making it hard to see the contribution of technology to the failure.

Failures don’t just happen at one shot. Failures accumulate.

People continuously create safety. Failure free operations are the result of activities of people who work to keep the system within the boundaries of tolerable performance. These activities are, for the most part, part of normal operations and superficially straightforward. But because system operations are never trouble free, human practitioner adaptations to changing conditions actually create safety from moment to moment. These adaptations often amount to just the selection of a well-rehearsed routine from a store of available responses; sometimes, however, the adaptations are novel combinations or de novo creations of new approaches.

People are what prevent failures.

Failure free operations require experience with failure. Recognizing hazard and successfully manipulating system operations to remain inside the tolerable performance boundaries requires intimate contact with failure. More robust system performance is likely to arise in systems where operators can discern the “edge of the envelope”. This is where system performance begins to deteriorate, becomes difficult to predict, or cannot be readily recovered. In intrinsically hazardous systems, operators are expected to encounter and appreciate hazards in ways that lead to overall performance that is desirable. Improved safety depends on providing operators with calibrated views of the hazards. It also depends on providing calibration about how their actions move system performance towards or away from the edge of the envelope.

Only when people have failed do they know how to prevent failure.




ever curious

Love podcasts or audiobooks? Learn on the go with our new app.

In The Loop — Chapter 26: AI, Machine Learning and Deep Learning

K-Fold Target Encoding for High Cardinality Features


Hong Kongese Identification with Deep Learning

Is there logic behind probability?

Image recognition for custom categories with TensorFlow

Nyoka — yet another snake in the Python Zoo

Thinking of Self-Studying Machine Learning? Remind yourself of these 6 things

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Hum Qing Ze

Hum Qing Ze

ever curious

More from Medium

Making life’s harsh realities work for you! Final series

Dan Ariely on decision making

How we made a shift from kotlinx.synthetics to Android View Binding

CS371p Spring 2022: Manasi Ramadurgum — Week 9