Notes from 13/7–20/7

COVID-19

I’m not sure how to feel about this at all

AI

Brilliantly done, just uses Gitbook and Youtube videos.

One of the most challenging issues is the difficulty to achieve high anomaly detection recall rate (Challenge #1). Since anomalies are highly rare and heterogeneous, it is difficult to identify all of the anomalies. Many normal instances are wrongly reported as anomalies while true yet sophisticated anomalies are missed.

Anomaly detection in high-dimensional and/or not-independent data (Challenge #2) is also a significant challenge. Anomalies often exhibit evident abnormal characteristics in a low-dimensional space yet become hidden and unnoticeable in a high-dimensional space. High-dimensional anomaly detection has been a long-standing problem. Subspace/feature selection-based methods may be a straightforward solution. However, identifying intricate (e.g., high-order, nonlinear and heterogeneous) feature interactions and couplings may be essential in high-dimensional data yet remains a major challenge for anomaly detection.

Due to the difficulty and cost of collecting large-scale labeled anomaly data, it is important to have data-efficient learning of normality/abnormality (Challenge #3). wo major challenges are how to learn expressive normality/abnormality representations with a small amount of labeled anomaly data and how to learn detection models that are generalized to novel anomalies uncovered by the given labeled anomaly data.

Many weakly/semi-supervised anomaly detection methods assume the given labeled training data is clean, which can be highly vulnerable to noisy instances that are mistakenly labeled as an opposite class label. One main challenge here is how to develop noise-resilient anomaly detection (Challenge #4).

Most of existing methods are for point anomalies, which cannot be used for conditional anomaly and group anomaly since they exhibit completely different behaviors from point anomalies. One main challenge here is to incorporate the concept of conditional/group anomalies into anomaly measures/models for the detection of those complex anomalies (Challenge #5).

In many critical domains there may be some major risks if anomaly detection models are directly used as black-box models. For example, the rare data instances reported as anomalies may lead to possible algorithmic bias against the minority groups presented in the data, such as under-represented groups in fraud detection and crime detection systems. An effective approach to mitigate this type of risk is to have anomaly explanation (Challenge #6) algorithms that provide straightforward clues about why a specific data instance is identified as anomaly. Providing such explanation can be as important as detection accuracy in some applications. To derive anomaly explanation from specific detection methods is still a largely unsolved problem, especially for complex models. Developing inherently interpretable anomaly detection models is also crucial, but it remains a main challenge to well balance the model’s interpretability and effectiveness.

I wonder if anyone has done this for Temasek/ Singapore

Blockchain

Basically employment contracts made smart?

Development

Brilliant and useful, will keep around

Education and Training

This existed since 2014? It looks absolutely brilliant!

Will they actually pull this off though?

Product

Covers quite a few fintech startups i’ve not actually come across before

Tools

An operating system for development

It’s a video on zoom? interesting idea

I’ve a feeling this will be useful when explaining effects

--

--

--

ever curious

Love podcasts or audiobooks? Learn on the go with our new app.

How to Visualize your Runners’ High with Python and Altair

Another Wave, the Same Mistakes

How to Perform Simple & Effective Customer Segmentation | A Walkthrough with Data from a…

Understanding language context

The impact of the COVID-19 pandemic on mental health and well-being of people living with a…

How to visualize data effectively and impressively?

EXERCISE IN THE DARK (L)ESS

Stock Market Valuation and Volatility with R

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Hum Qing Ze

Hum Qing Ze

ever curious

More from Medium

Mykplan Adp Sign In — How to Retrieve Forgot Mykplan Adp Login Password

Making life’s harsh realities work for you! Final series

Dan Ariely on decision making

Librarylife #6