Is it time to start looking out for a new job?

Career Decision - Next ExitI recently happened to read this post from the Spend Matters blog. Blog Post talks about three questions one should ask themselves regularly to decide whether it is time to look for a new job.

1. Do you feel you are personally learning and developing in your current role? Are you gaining new skills, developing your capability, becoming more expert – with the caveat that it really helps if these are in some sense transferable skills and knowledge.

2. Are you progressing in your career, moving forward from a seniority and / or financial point of view, with a trajectory that is heading in the right direction? (This assumes you have some ambition – not everyone does have, I realize).

3. Do you enjoy what you are doing – is it a good place to work, with decent “hygiene factors” and a bit more – which might be anything from pleasant colleagues and working environment, a commute that is manageable, technology that works, social events …

Excellent set of questions. You can find the original post here. 

This one is my favorite on this topic. Probably tied to all the 3 questions above.

To be happy and be fulfilled at work, people want to feel they are advancing, getting things done, and making an impact. But it’s not enough to simply to receive a pat on the back and a word of encouragement. Rather, we respond much more positively to feedback from the work itself. When we have achieved a goal like closing a sale, writing code that passes the test harness and is pushed to production, releasing a new feature that a million users touch every day, our happiness at work blooms.

Source: Managing for Progress

An Excellent book on this topic : The Progress Principle

Teresa Amabile’s talk at Google

In the end… it is not about the nice office buildings, additional perks etc. It is about the job itself. It is about the people you interact with on a daily basis and deal with.

The Successful demos at the end of every 2 weeks, Production Releases, a good solution to a complex problem, providing a solution using a new technology stack, number of first calls to potential prospects, a good sales pitch to a new prospect, a new customer win on a regular basis, hiring a good candidate, coming up with a new product offering… (A few items from my list). All of these contribute to the small wins part. If you dont have them as part of your day to day job or you dont see the number of small wins…. may be it is time!

Happy Learning!

Advertisements

Scaling data operations with in-memory OLTP

Data has become the center of our universe in modern digital world. Applications are designed to store and collect more and more data. Companies are looking to integrate and analyse the data to generate insights and take actions.

Data is a precious thing and will last longer than the systems themselves ~ Tim Berners-Lee

Can an existing relational database scale with high ingestion rates, improved read performance?Database

In-Memory OLTP seems to be the direction forward. This is considering your existing technology investments. Of course if the company is open to change technology there would be more options.

Found couple of very good articles posts related to SQL Server in-memory OLTP. Looks like SQL Server 2016 has fixes to most of the issues with in-memory OLTP.

I just think it is an amazing technology and if we can use it in the right way, will definitely yield great results for your customers.

Introducing SQL Server In-Memory OLTP
https://msdn.microsoft.com/en-in/library/dn133186.aspx
https://www.simple-talk.com/sql/learn-sql-server/introducing-sql-server-in-memory-oltp/
http://blog.sqlauthority.com/2014/08/08/sql-server-introduction-to-sql-server-2014-in-memory-oltp/

The Use Cases for SQL Server 2014 In-Memory OLTP
http://sqlturbo.com/the-use-cases-for-sql-server-2014-in-memory-oltp/

SQL Server In-Memory OLTP Internals Overview
https://msdn.microsoft.com/en-us/library/dn720242.aspx

The Promise – and the Pitfalls – of In-Memory OLTP
https://www.simple-talk.com/sql/performance/the-promise—and-the-pitfalls—of-in-memory-oltp/
https://msdn.microsoft.com/en-us/library/dn246937.aspx

SQL Server 2016 : In-Memory OLTP Enhancements
http://sqlperformance.com/2015/11/sql-server-2016/in-memory-oltp-enhancements

Speeding up Business Analytics Using In-Memory Technology
https://blogs.technet.microsoft.com/dataplatforminsider/2015/12/08/speeding-up-business-analytics-using-in-memory-technology/

Dynamic Data Masking in SQL Server 2016
http://www.codeproject.com/Articles/1084808/Dynamic-Data-Masking-in-SQL-Server
https://blogs.technet.microsoft.com/dataplatforminsider/2016/01/25/use-dynamic-data-masking-to-obfuscate-your-sensitive-data/

Happy Learning!

Software Architecture, Customer Success

Happened to Watch couple of good videos last week on Software Architecture, Design and Customer Success.

How the World Wide Web just happened – Tim Berners-Lee
https://www.youtube.com/watch?v=yF5-6AcohQw
Great Session. Talks about the importance of being in the right place and the right time.

Mary Poppendieck (Poppendieck.LLC) – The New New Software Development Game: Containers, Micro Services
http://m.ustream.tv/recorded/61477219?rmalang=de_DE
Complexity grows non-linearly with Software size. Software size continues to grow so software complexity will continue to grow even faster. She explains what can we do about the complexity?

A summary of this talk is available here
http://highscalability.com/blog/2015/4/27/how-can-we-build-better-complex-systems-containers-microserv.html

Zen and the art of Customer Relationships
https://www.youtube.com/watch?v=G_2UP4-J7Vc
I loved the Zen and the Art of Customer Relationships presentation from Zen Desk. Awesome Presentation!
Pointers for building long lasting relationships

  1. Don’t overestimate your importance in your customers life
  2. Consider the entire customer experience
  3. Recognize the right relationships and adapt
  4. Be something actual humans can relate to
  5. Be Transparent
  6. Empower your best people to do what’s best
  7. Put a face to your customers

Framework to Build a Killer Customer Success Scorecard
https://www.youtube.com/watch?v=lhx06h8RZ3Q
Another Fantastic presentation from the trenches. A good overview around how to define Customer Success and what are the metrics to monitor (Customer, Financial, Practice and Inter-team)

Building the Customer Success Management Team
https://www.youtube.com/watch?v=XIx5HhfG56w
Happy Learning!

Bye Bye 2015! HNY 2016!

Another year has passed. We are towards the end of the year and It is time to reflect on 2015. Every year, there will be always good and bad things. This year is no different in that way for me. May be the percentage of not so good was more in 2015 😦

Good things to remember in 2015

  • Towards the end of 2014, had an opportunity to work on creating a SaaS based Real time streaming solution for a large security company. 2015 started with the win of this project and what a way to kick start the year!
  • Got an wonderful opportunity to work on integrating data from various marketing and digital channels and create a data lake for the world’s leading postal company.
  • Got an opportunity to work with a team to implement in-memory (Hana based) business intelligence platform for the world’s largest ketchup manufacturer. Worked with their Enterprise Architecture team to create a security blueprint and IDM implementation roadmap.
  • Did a fair amount of work with DevOps in 2014. Got a great opportunity to present/demonstrate DevOps capabilities to a great set of technical folks. Helped in winning the deal.
  • Got opportunities to understand and work with teams to implement HOLAP and Stream analytics.
  • Got exposure to Customer Service in an Enterprise SaaS world.
  • Understood the difference between Reporting and Dashboards (in a hard way).
  • The way I looked at SaaS architecture changed dramatically. Now, I look at the Maturity models based on the business landscape.

Personally, the start-up bug started biting me towards the end of 2014. We were close to starting a Services business on our own. Explored couple of Product ideas as well. Took one of the product idea and detailed it to a great extent. Did a fair amount of Customer Discovery and Validation exercise.

What I initially thought as a vertical add-on, eventually became a mammoth horizontal platform idea. Realized that it would take at least 3 to 4 years before can do anything with it. Dropped it after living & breathing the idea for almost 3 months!

Learnt a lot in the whole process.  Though it was a failed attempt, at least learnt what it means to take a hypothesis, visualize, conceptualize and start something 🙂

If you don’t code on your own, better don’t get into the start-up thought  process.

Thanks to Neelam, Subhajit, Sudhakar, Ashish, Gayathri and Sendhil for their help in the validation exercise.

Made some not so good (crazy) decisions with my career this year! Though it was not a great decision, at least i don’t have the thought that I haven’t tried anything new anymore.

No matter how many mistakes you make or how slow you progress, you are still way ahead of everyone who isn’t trying ~ Tony Robbins

From a technology front, 2015 was an year full of Data related projects for me. My understanding on this space has become much better in 2015.

Overall, 2015 was a decent year! An year filled with career adventures, self-disruption, lots and lots of learning’s!

Hope 2016 will be a better year!

Happy New Year 2016!

Wishing you all a Very happy new year!

May the NEW YEAR bring you GOOD HEALTHPEACE and HAPPINESS.

Image Source: http://happynewyear2.com/tag/happy-new-year-2016-greeting-cards/

“Data is long-term, Applications are temporary.”

Think data first. Data is long-term, applications are temporary. I recently happened to read this in one of the blog post. I couldn’t agree more. Data remains one of the most strategic projects for most of the companies.

Every fifth person you talk to, every other start up you come across and job postings has something or other to mention about data, analytics etc. But, when I speak to the guys whoever I come across in my ecosystem, lot of guys think it is only doing cool stuff in R.

Big data is like teenage sex: everyone talks about it, nobody really knows how to do it, everyone thinks everyone else is doing it, so everyone claims they are doing it.

If someone is an application developer for the last 10 years, can he/she suddenly become an expert in statistics and become an expert in Algorithms? Suddenly you start calling yourself a Data Scientist? May be… Nothing is impossible. But if that’s what is your passion you wouldn’t be an application developer for the last 10 years. Right?

Is there anything else one can learn and contribute in the data world? Thought of sharing couple of valuable links which can give you a very good idea on the various aspects and where one can fit in.

#1 Will Balkanization of Data Science led to one Empire or many Republics? Via http://www.kdnuggets.com/2015/11/balkanization-data-science.html
#2 Becoming a Data Scientist via http://nirvacana.com/thoughts/becoming-a-data-scientist/
#3 Difference between Data Engineering and Data Science via http://www.galvanize.com/blog/difference-between-data-engineering-and-data-science/
#4 The world of data science: Who does what in the data world? Via http://cloudtweaks.com/2015/11/booming-world-data-science/matrix-1013612_640

Data is one of the hottest stack right now and it is growing at a crazy speed. It would be extremely difficult for any single individual to cope up with this change unless one’s basics are right.

Once you have the basics right, it is about Meta learning and evolving from there.

Working with various large scale data related projects for the last 15 months, following is my high level list of items one need to know to have a reasonable understanding of data (Big/Small). This list is no specific order. 😦

General A Basic overview of what is Descriptive, Diagnostic, Prescriptive, Predictive and Cognitive Analytics? Understanding of the concepts and difference
Data Warehouses
  • OLAP VS OLTP
  • Dimensional Modelling (Star Schemas, Snowflake Schemas)
  • Difference between Multi-Dimensional, Relational, Hybrid
  • In-Memory OLAP
No SQL Databases
  • CAP Theorem
  • If you are from application development, this is where the most important change would be. So far, you would have dealt primarily with Key-Value stores and Document Stores. For Analytics purpose (Write Efficient), it is important to start understanding column databases (E.g.: Cassandra) and Graph (E.g.:Neo4J). This is again a big shift from what you would have done as an application developer. Spend some time on it.
  • In-Memory databases in general.
  • Apart from Cassandra and Neo4J, get an understanding of what MemSQL offers. Yes, it is MemSQL and not MySQL J seems very impressive.
Outside EDWs
  • MPPs/PDWs – Difference between traditional EDWs and MPPs?
  • DWH on cloud AWS Redshift, Azure SQL Data Warehouse
Data Mining
  • What does it mean?
  • Data Mining Algorithms
Hadoop
  • Hadoop and Various Hadoop Components
  • When to use Hadoop?
  • Parallelization and Map Reduce Fundamentals
Outside Hadoop
  • Difference between Hadoop, Spark and Storm (I personally prefer SPARK. RDDs give me the same comfort what I had with ADO.NET)
  • When to use Hadoop/Spark/Storm over MPP?
ETL
  • Data Munging/Wrangling
  • Scrubbing
  • Transforming
  • Reading and Loading Data
  • Exception Handling
  • Jobs/Tasks
Real time Analytics Working with Stream: Real time Analytics is something everyone talks about. But without understanding what it means by Stream processing you will never be able to figure out this.
From an application background

  • Reactive Architecture (Responsive, Resilient, Elastic and Message driven)
  • Understand the difference between an Event and a Transaction.
  • Event Processing(CQRS, Actor Model[Akka], Complex Event Processing)

If you don’t understand the above, then it would be difficult to move forward. Spend time on these before moving forward to other items
Messaging/Data bus

  • Kafka

Processing Streams

  • Spark/Storm

Lambda Architecture

Machine Learning Machine Learning

  • Difference between Data Mining and Machine Learning
  • ML Algorithms

Couple of very good posts to read in this
Machine Learning for Programmers: Leap from developer to machine learning practitioner via http://machinelearningmastery.com/machine-learning-for-programmers/
What Every Manager Should Know About Machine Learning via https://hbr.org/2015/07/what-every-manager-should-know-about-machine-learning
Most of what we are doing can be achieved at some level using Excel Analytics Data Pack. In fact, I would say Excel is the most powerful tool out there.

Recommendation Engines
  • Collaborative Filtering
  • Content-based Filtering
  • Hybrid

Once you are clear with the concepts start implementing using Apache Mahout

Communication Protocols
  • JSON, AVRO, Protocol Buffer, and Thrift: If you are from application development – you would have used JSON extensively. It is time to understand the other ones as well. I keep arguing this with my friend Sendhil (IMO, AVRO seems to be the way to go – where things are evolving and need for self-documentation – Cowboys Friendly).
Time Series
  • Modelling
  • Databases (OpenTSDB)
  • Forecasting
  • Trend Analysis
Modern day HOLAP Engines
  • Apache Kylin (My favourite at this point)
Data Visualization Self-Service is the Mantra here. Read this article: Data Scientists Should be Good Storytellers

Most of the people in an organization cannot understand the outcome of analytics, however they do need the proof of analysis and data. Data storytellers incorporate data and analytics in a compelling way as their stories involve real people and organizations” via https://dzone.com/articles/data-scientists-should-be-good-storytellers

  • How to represent data (Graphs/Charts)?
  • Excel Power Pivot/ Power BI (Polybase)
  • Lumira
  • D3.js
Deep Learning Though it may or may not be important at this point, try to understand what is deep learning. Read this : Deep Learning in a Nutshell: Core Concepts via http://devblogs.nvidia.com/parallelforall/deep-learning-nutshell-core-concepts/
Data Lake One of my favorite topic and something I learnt after burning my hands is with data lake

  • Understand what Data Lakes mean? Why do you need one? How to build a data lake on your own?
  • Extract Load and Transform (ELT)
  • ELT vs ETL

Read this: https://azure.microsoft.com/en-in/solutions/data-lake/

Language Though there is a bunch of things to do with Python, R, Java etc. My choice is Scala (I love the way the language allows you to express. Wish someone can afford me as a developer again J)

If you have a good grasp on above, then it is time for you to figure our when to use what (Creating Solutions).

 “If all you have is a hammer, everything looks like a nail”

Read this:  The Ethics of Wielding an Analytical Hammer via http://sloanreview.mit.edu/article/the-ethics-of-wielding-an-analytical-hammer/

Data is having an impact on business models and profitability. It’s hard to find a non-trivial application that doesn’t use data in a significant manner ~ Ben Lorica, O’Reilly Media

Ok, this looks like a large list. Where do I start?

  1. Focus on the basics. Get a good overview of the ecosystem
  2. Decide your area of specialization.
  3. Focus on your specialization and build skills.
  4. Iterate and change course as required.
  • If you are more than 10 years of experience, understand the business situation and figure out when to use what. May be pick 1 or 2 items and start implementing in your environment.
  • If you are less than 10 years of experience, pick up a scenario and try to implement this and see if it makes any business sense.

What I have not covered in the list? I haven’t gone into the details of

  1. Hadoop Ecosystem and components (Pig/Hive etc.)
  2. Algorithms
    1. Nearest Neighbour
    2. K-Means Clustering
    3. Linear Regression
    4. Decision Trees etc.
  3. R in detail
  4. Infrastructure
    1. Env Setup
    2. Zookeeper, Yarn, Mesos
    3. Replication
  5. Vertical Industry Solutions
  6. Operational Systems (like Splunk)
  7. Data Governance

I keep hearing/seeing people who have never seen more than 1 GB of data saying that they do Big Data Analytics. Don’t learn or do something for the sake of doing it.

There is no short cut to a place worth going.

My favorite books on this topic.

If you want to know more about what I am learning, you can follow me in Twitter

Happy Learning!