OpenAI GPT-2: Understanding Language Generation through Visualization
GPT’s downfall was that it was pre-trained using traditional language modeling, i.e., predicting the next word in a sentence. In contrast, BERT was pre-trained using masked language modeling, which is more of a fill-in-the-blanks exercise: guessing missing (“masked”) words given the words that came before and after.
Interesting to see that GPT generates biographies with a bias on the perceived ethnicity and sex associated with a name.
The blog basically used Tensor2Tensor to visualise how GPT-2 made decisions.
BERT makes use of Transformer, an attention mechanism that learns contextual relations between words (or sub-words) in a text.
Before feeding word sequences into BERT, 15% of the words in each sequence are replaced with a [MASK] token. The model then attempts to predict the original value of the masked words, based on the context provided by the other, non-masked, words in the sequence
Next Sentence Prediction
the model receives pairs of sentences as input and learns to predict if the second sentence in the pair is the subsequent sentence in the original document. During training, 50% of the inputs are a pair in which the second sentence is the subsequent sentence in the original document, while in the other 50% a random sentence from the corpus is chosen as the second sentence.
You can also finetuned BERT to some other tasks such as NER, Question answering and classification. Okay but it takes 4 days to train.
Train a classifier model using open 3D model datasets to predict what sort of 3D model is being built, then recommend them to the designer. I suppose a similar concept can be used when it comes to other things that require some sort of prompting.
I wouldn’t exactly call these decision models per se. More of conceptual frameworks. However, they are really high level and requires some unpacking to actually answer the question.
ScrapedIn — to search LinkedIn
Simple tool to exploit people who, in their folly, openly declare they have top secret security clearance in order to get jobs in security companies.
The 2019 Web Developer Roadmap — A Visual Guide to Becoming a Front End, Back End, or DevOps Developer
Okay maybe it’s time to do a deep dive into this. why not eh?
man [COMMAND] to view the arguments supported
cat [FILE_PATH] opens up the file
head [FILE_PATH] or
tail [FILE_PATH] to see only the top or bottom lines
wc -l / -w /-c [FILE_PATH] that lets you count the lines
-l , words
-w or characters
grep "[SEARCH_STRING]" [FILE_PATH] printing all lines that contain your search string
| piping data from one command to the next
sort -t "[DELIMTER]" -k [COLUMN] -r -n [FILEPATH] sorts contents of the file according to the delimiter, the column to sort on,
-r is for descending sort (it sorts by ascending by default) and
-n for numerical sorting
uniq used when you pipe data from a
cut -d "[DELIMITER]" -f [COLUMNS] [FILE_PATH] to view specific columns
tr "[CURRENT]" "[REPLACEMENT]" replace characters
find . -name "A*.csv" this is a search command that lets you find all the files strating with A. * is a wildcard
> [FILE_NAME.txt]write into another file
Great way to explain a typical probability question. All about using estimators. Mentions your Maximum Likelihood Estimate (MLE) and your Minimum Variance Unbiased Estimator (MVUE)
Foucault suggests that there is no fixed, ahistorical notion of human nature, as posited by Chomsky’s concept of innate human faculties. Chomsky argues that concepts of justice are rooted in human nature and reason, whereas Foucault rejects any such universal basis for a concept of justice.
Foucault is really radical in this aspect. Justice, from what I understand, comes from human reason. Only then can we decide what constitutes a just act. Chomsky was quite insistent that a just society must promote the truest aspects of human nature. But, human nature might just be impossible to define. Thus, socity should promote an ideal concept of human nature but still be open to changes. Now that, fundamentally will be very difficult due to the way institutions are created to enforce these idealised notions of human nature itself.
We are developing a Facebook Messenger chatbot that helps migrant to:
Connect with a community and social support network in their arrival city.
Maintain, manage, and/or get new documentation or registration upon arrival.
Match employment based on their skills.
Affordable housing information.
Interesting concept. It automates a lot of these painful tasks that are highly consuing for non locals.
Quick dive into SEO.
So that’s what SEO features are for. It’s just a way to optimise your site for Google’s crawlers.
Decision-making framework that can be adapted, something useful to take a look?
- Transparency, which empowers employees to make their own decisions
- Intellectual Rigor, which helps the company uncover innovative ideas
- Radical Candor, which helps us to grow fast
Makes one responsible for their own decisions and rewards initiative. But allows one to receive frequent feedback so refine these said ideas.
Hindsight biases post-accident assessments of human performance. Knowledge of the outcome makes it seem that events leading to the outcome should have appeared more salient to practitioners at the time than was actually the case. This means that ex post facto accident analysis of human performance is inaccurate. The outcome knowledge poisons the ability of after-accident observers to recreate the view of practitioners before the accident of those same factors. It seems that practitioners “should have known” that the factors would “inevitably” lead to an accident.2 Hindsight bias remains the primary obstacle to accident investigation, especially when expert human performance is involved.
Perfect way to analyse incidents. Qualifies the concept that hindsight is 20/20
Actions at the sharp end resolve all ambiguity. Organizations are ambiguous, often intentionally, about the relationship between production targets, efficient use of resources, economy and costs of operations, and acceptable risks of low and high consequence accidents. All ambiguity is resolved by actions of practitioners at the sharp end of the system. After an accident, practitioner actions may be regarded as ‘errors’ or ‘violations’ but these evaluations are heavily biased by hindsight and ignore the other driving forces, especially production pressure.
So… ultimately organisations create an illusion of order. It allows for confidence to act rather than to create the parameters for acting.
Change introduces new forms of failure. The low rate of overt accidents in reliable systems may encourage changes, especially the use of new technology, to decrease the number of low consequence but high frequency failures. These changes maybe actually create opportunities for new, low frequency but high consequence failures. When new technologies are used to eliminate well understood system failures or to gain high precision performance they often introduce new pathways to large scale, catastrophic failures. Not uncommonly, these new, rare catastrophes have even greater impact than those eliminated by the new technology. These new forms of failure are difficult to see before the fact; attention is paid mostly to the putative beneficial characteristics of the changes. Because these new, high consequence accidents occur at a low rate, multiple system changes may occur before an accident, making it hard to see the contribution of technology to the failure.
Failures don’t just happen at one shot. Failures accumulate.
People continuously create safety. Failure free operations are the result of activities of people who work to keep the system within the boundaries of tolerable performance. These activities are, for the most part, part of normal operations and superficially straightforward. But because system operations are never trouble free, human practitioner adaptations to changing conditions actually create safety from moment to moment. These adaptations often amount to just the selection of a well-rehearsed routine from a store of available responses; sometimes, however, the adaptations are novel combinations or de novo creations of new approaches.
People are what prevent failures.
Failure free operations require experience with failure. Recognizing hazard and successfully manipulating system operations to remain inside the tolerable performance boundaries requires intimate contact with failure. More robust system performance is likely to arise in systems where operators can discern the “edge of the envelope”. This is where system performance begins to deteriorate, becomes difficult to predict, or cannot be readily recovered. In intrinsically hazardous systems, operators are expected to encounter and appreciate hazards in ways that lead to overall performance that is desirable. Improved safety depends on providing operators with calibrated views of the hazards. It also depends on providing calibration about how their actions move system performance towards or away from the edge of the envelope.
Only when people have failed do they know how to prevent failure.