Book Review: I Am Malala

I Am Malala: The Girl Who Stood Up for Education and Was Shot by the Taliban by Malala Yousafzai
My rating: 4 of 5 stars

For a 15 year old to have such insight, drive and the grit, it is astonishing to see Malala just going ahead with her business and not give up. Having faced such hardships right from the environment in Swat valley to being shot, Malala is a symbol of courage and I have garnered a lot respect for this nobel prize laureate.

Regarding the book, I started reading this book to gain some real insight in that part of the world and I am glad I started with this masterpiece. The book covers a bit of history of Pakistan to begin with. Then Malala goes on to talk of her childhood friends and family. Her grip on penning down what she feels of her parents and how they look at it is again commendable. Its quite inspiring to see Malala and her father fighting for the fundamental human right of girls education knowing there is a threat looming all the time. You see how both of them are not afraid come what may but when it comes to the other, they do get protective. Amidst all the life story, Malala presents an amazing account of a father-daughter relationship. Malala seems to have a lot of respect for her culture and religion at the same time she is not willing to be fooled by the impositions of gender inequality. Reading this book is a reminder of how much we take for granted things around us. I am sure I will appreciate my freedom a tad bit more now. I hope and wish her dream of getting back to Swat valley and the vision she carries of her motherland does come true. The world needs more Malalas. Looking forward to reading more about her life as she carries on the fight she has taken.

View all my reviews

Monitoring Java based Applications: Metrics+Graphite+Grafana

As developers when we write a piece of code we try our best to make sure we have carefully chosen the best algorithms, the data structures are carefully tuned to match the situation at hand and the piece of software we wrote just works out of the box. Unfortunately life is not ideal and there are some hiccups that are bound to hit us however smooth the ride be. Some nasty bugs will inevitably show up on the production server as the software is used for a length of time. The best one can do then is to monitor the software being used in production environment, anticipate the problems before it hits the user and hampers their experience.

This post will talk about such a stack for monitoring java based applications involving:

1. Dropwizard Metrics
2. Graphite and
3. Grafana

We will use dropwizard metrics to instrument our Java code and push the data to graphite. Graphite will store our time-series data meaning values such as latency, performance, throughput, counts etc corresponding to say our APIs. Though Graphite renders graphs built on top of the time-series data as and when required, Graphite on its own doesn't provide the best visual experience and dashboard properties. We will thence use Grafana to build and view our dashboards. So the pipeline on a high level will be this:

Here is the documentation of dropwizard metrics listing its various capabilities. Their documentation is not that great, in my opinion, but workable. The good news is there is also a spring library that provides annotations you can use directly to enable instrumentation. Just add an annotation on top of a method and thats it. Here is an example:

@Metered(absolute = true, name= "RuntimeError")

public ErrorMessage handleThrowable(HttpServletRequest request, Throwable t) {

            ………..

            }

As you see here, there is an annotation (@Metred) which is responsible for instrumenting the following method- handleThrowable(...). The metred annotation gives the timed average meaning number of errors in a unit of time. As simple as that! The code in bold is all you need to write to instrument the following method, for instance.

Next, we need to tell our metrics where Graphite resides so data can be sent to Graphite. Here is how you configure this. The prefixedWith() attribute defines how Graphite will show this. Here is a screenshot from my Graphite dashboard showing the tree-based structure that we just set.

On the right of this pane shown above you ca see the graph you select. The visual features in Graphite is rather limited so we will forward the Graphit input to Grafana. Here is a video introducing Grafana interface.e

Make your dashboards the way you wish to see your graphs and keep monitoring to stay ahead of the bugs!

Log Archiving with Logstash

A crash, a panic attack and we rush to the logs hoping it comes to rescue. Alas there is so much content in these logs that finding the relevant content itself is a herculean task. It is thus a good idea to think upfront and brace yourself to handle the panic situation better. Keep all the logs ( application logs, database logs etc) nicely organized in a tree structure all of them being kept at one place. More technically, it is a good idea to archive your logs and have a way to make sense of the information hidden in logs. I was recently faced with such a situation and ended up making an ELK based stack.

I started with syslog-ng. Syslog has been around for sometime so it gave me a sense of stability and good community around it. Unfortunately syslog is a bit outdated too! The next thing we considered was logstash. Logstash along with Elasticsearch  and Kibana forms a powerful stack, popularly known as the ELK stack.

For this blog post, we will limit our scope and stick to logstash part of the ELK stack. We will make an archive which collects logs from different web and database servers and archives them all in one place. The intelligence and visual analysis are left to Elasticsearch and Kibana respectively and thence not part of this post. Following is a very high level architecture I came up with:

We are going to work with two pieces of software here:

1. logstash-forwarder
2. logstash

We will install logstash-forwarder on the clients (our applications that generate logs). This is an agent that sends the logs to a central archive server. Note that logstash-forwarder needs to be installed at EVERY client. We will next install logstash on the archiving server. This will receive logs from every client and archive it in a format that you can define. Here is what I came up with for my archive structure.

I like to archive by time and then the log source as depicted in the picture above. Now that we are done with sorting the architecture and our logging format we need to next configure our clients and the server. Digitalocean has done a wonderful job of explaining the installations and configurations so I will leave you with this article. You can skip the parts involving Elasticsearch, Kibana and ngnix.

Once you are done with archiving make sure you have a way to periodically clean logs from the application servers, if at all you store last 1-2 days of logs there too. For this I set up a cron job that cleans up logs older than a week in my case.

Book Review: I Too Had a Dream

I Too Had a Dream by Verghese Kurien
My rating: 5 of 5 stars

With autobiographies I sometimes don't understand how deeply you trust what you read, how you verify the facts and what alterate sources to trust. Thence I am generally not a big fan of reading autobiographies even though I love biographies. Nevertheless I am happy I picked up this book for this is an honest account on Mr Verghese Kurien. You can feel the empathy for customers in his words. You can see his intent and passion for cooperatives.

The book begins with early life of Mr Kurien and how through an accidental turn of events Mr Kurien was forced into diary engineering and then working in Kiara, how Sardar Vallabhbhai Patel and Tribhuvandas affect his life and vision and how he fought through all the politics and bureaucracies to make one of the biggest cooperatives in India. His constant emphasis on cooperative vs corporate is a food for thought. Though he blatantly criticises corporate and sometimes you tend to disagree with him this book presents a good view of the two worlds- cooperative and corporate Further, the obsession of author for the delicate balance between the benefits of both producers (farmers) and consumers of milk (or other agricultural products) is very compelling and provides for an engaging lession on marketing.

Amul and Dhara as brands stand as tall success to prove the success he achieved. The real success of Mr Kurien is way above these though. This book thus classifies as a must read for the impeccable writing plus insights on cooperative strategy and marketing.

View all my reviews

MobileSpark conference 2015: Impressions on Day 1

MobileSpark is a two day conference and MobileSpark 2015 happened last Friday and Saturday (August 6 and 7, 2015). I attended the complete event and here I put down some impressions from the event. I tried to lure some of my friends to join me, also thinking of the hackathon but no one did. Perhaps the agenda did not look as compelling to them. Fair enough, I thought if I am interested I should simply go. The venue was far, around 20 kilometers from my house but I did manage to attend both the days and all the events. Lets get on with the nitty-gritty details now with the two days.

The event was due to begin at 9:30 in the morning, it didn't. Until 10:00 there was no sign of the first talk. Then appeared an over-enthusiastic and loud anchor. The first couple of sessions were not very engaging enough to change the perception either. My enthusiasm dropped a bit. I started to feel my friends who decided against attending this conference were right. I was going to find it a bit tough to spend two days here. Thankfully it did pick up and I sort of enjoyed being there at the end of it all, specially the second day. Lets iterate through the sessions one by one.

1. The first session was by Ashish Agrawal, CTO, Micromax & YU. He began crunching some numbers to show that Yureka OS had already made some fans and further went to essay the importance of community contributions in building their suite of software. Ashish thanked them and invited more contributions.
2. Next was a talk by Arvinder Gujral, Director, BD, Twitter APAC. He talked of Twitter and the apponomics of today essaying how startups can leverage twitters's SDKs (Fabric, Crashlytics etc) and user-base to gain traction.
3. Next, it was Varun Khurana, CTO, Grofers talking of e-commerces of today namely, Flipkarts and Snapdeals and how hyperlocal was the new thing on the block. Indeed there are a ton of hyperlocals and even those that only aim to facilitate hyperlocal startups by facilitating parts of the business. Grofers had a big presence on this conference in general and even stood out as one of the sponsors of the event.
4. Next, we had Rajat Upadhyaya, VP, Engineering Urban Ladder talking of the space Urban Ladder is in and why the multiple-app strategy they have built makes sense. In their case they have three customer facing apps.
5. Following this was the best presentation I have come across in some time. Girish Mathrubootham, Founder & CEO, Freshdesk came across to talked on how to build a global product and how to assemble a rockstar team for the same. Girish's was surely the best presentation while he talked about how Freshdesk began and his philosophies on the hierarchy structure, recruiting, managing his employees etc. It was quite thought provoking. As Girish finished his session he got an standing ovation from the crowd and deservedly so.
6. Lately I have become a fan of the culture InMobi has and I was really looking forward to the next talk but there was not much I could collect from this session. The coffee breaks were discarded in respect of time and perhaps had their obvious impacts on me too. After all, we had Chaipoint serving 24*7 their awesome Chai.
7. Going forward to the vision of OYO rooms. They target the tier-2, 3 hotels and have a really compelling vision to facilitate the hotel experience. Starting from helping both sides through technology to being empathetic about the infrastructure and maintenance of these hotels to the on-boarding and staying experience for guests they have a grand vision and I really do wish they fulfill it. There is a clear space and I am sure this will simplify the lives of a lot of people.

I skipped through the next couple of sessions to grab some food and have a look at all the stalls various start-ups had set up. Here is a list: Hyperverge, Practo, Chaipoint, Yu, Grofers, Freshdesk, Flipkart, Oyo rooms, Process9, Affle, Urban ladder, Freecharge, Appvigil, Jobspire, Street smart, Babajobs, Playo, Makemyroti, Tapwisdom, Klever kid, Notifie, Reverie, Collection point, Msg 91, Mobignosis, Bizenabler, Localoye, Qykapp, Giftxoxo, Driveu, Unbxd, Furlenco. 

In a later post, I will capture some ideas from this list that caught my eye. For now lets move on to the afternoon session consisting mostly of panel discussions. There were three of them:

1. Mobile & local commerce: your neighbourhood at your fingertips - Varun Khurana, Grofers; Aditya Rao, LocalOye; Navneet Singh, Peppertap; Samar Singla, Jugnoo
   
   We have talked of Grofers earlier. Basically it is a hyperlocal grocery delivery startup. You can order groceries on their mobile app and get it delivered like BigBasket, peppertap etc.
   There are so many startup delivering products at the doorstep, not so many that deliver service. LocalOye is one such business. They claim to have a big checklist to get a service provider (for example- a carpenter, plumber etc) on board and they manage the whole experience of such a person coming to your house and delivering the service with an acceptable customer satisfaction. I find it a bit tough to agree that they can control their service guys but I will be really happy to be proved wrong.
   Peppertap is another hyperlocal grocery delivering startup and I really don't know yet how their business is different from Grofers but they do operate in multiple cities and promise to deliver within 2 hours.
   Jugnoo is an on demand auto rickshaw app connecting passengers wishing to travel from one place to another within a particular city  and strives towards making Auto- Rickshaws more affordable and user friendly. Being in Bangalore its almost un-imaginable that will ever happen. Samar also mentioned that they could become a partner to the other hyperlocal businesses such as Grofer, Peppertap etc. Jugnoo has also expanded lately jugnoo meals and jugnoo fatafat.
   
   The discussion revolved around the future of hyperlocal businesses, how big the market is, how many of them will survive in the years to come etc.
2. Foodtech: delivering all parts of the equations - Shashank S, Dazo; Harsha T, MasalaBox; Karthik V, MomoE; Arpit D, Roadrunnr; Aruj G, Bhukkad
   
   Foodtech is another buzz word today and one of the most obvious market along with hyperlocal. Dazo aims at making it easy to select food and offers variety too. They have a curated list of a small number of items on the menu everyday so people people don't fret over choosing what to eat.
   MasalaBox has been serving home made (sort of) food in Kochi for roughly six months and they recently moved to Bangalore too.
   MomoE simplifies the payments and Bhukkad prides at being the one that serves healthy food along with varierty and good taste.
   
   There were issues around packaging discussed and everyone on the panel agreed that packaging is a big deal. They do want to pack completely non-plastic but they don't see a practical, affordable and easy to adopt alternative. MasalaBox has tried being on that front but failed to scale and maintain the packaging standards. So this space is up for disruption.
3. The leaky bucket : Balancing Growth and Retention
   
   To me this came as the most engaging session. The idea was that number of downloads to judge the success of a mobile app is a wrong metric. One needs to see how many of them are repeat customers, how many of them uninstalled the app before or after first use, if one came back to install it after an uninstall etc. The panel discussed some common reasons for uninstalls and how to attend to such issues, how to increase engagement on the app etc.

Book Review: The Accidental Prime Minister : The Making and Unmaking of Manmohan Singh

The Accidental Prime Minister : The Making and Unmaking of Manmohan Singh by Sanjaya Baru
My rating: 5 of 5 stars

The review for this book can be divided in two parts. One, the authors presentation which deserves full marks. The author seems in complete freedom to express his views. The biggest worry for a book of this kind is that the author could be curbed by the high profile officials, which doesn't seem to be the case here. The books gives a balanced view on the two terms served by Dr Manmohan Singh. This is a must read for someone who has the slightest interest in Indian politics. For others too it contains a good account on the leadership, politics in general, the importance of image building etc.

Moving to the second part of review, if you have ever criticized Manmohan, you must read this book. The author allows the reader to form his own opinion quoting instances from his experience. And in my view Manmohan is an idol. Yes, he did loose out on his image in his second term quite miserably and deserves some flak for that too. But he deserves an immense respect for his integrity, patriotism, selflessness and for just being who he really was. As you read the book you feel for him. You wish he had some more power, you wish he had established his authority a bit more and lastly you wish he was on the right side. To end with I wouldn't call Dr. Singh the accidental prime minister but a great intellect just fallen in the wrong place at the wrong time. India will do well to home more people like him in politics.

View all my reviews

Book Review: Zero to One: Notes on Startups, or How to Build the Future

Zero to One: Notes on Startups, or How to Build the Future by Peter Thiel
My rating: 3 of 5 stars

The book is good in patches. To start with, the book did not interest me. I somehow continued and then started finding portions which were really engaging. You see the occasional brilliance of Peter Thiel as a really experienced entrepreneur. He comes across with some really valuable insights. Unfortunately, he wavers easily too. I kept trying to guess who the target audience of this book is. Perhaps the author tries to put in something for everyone which obviously backfires. Because this book is not very engaging I wouldn't recommend it to someone who is not very enthusiastic about the topic but I really look forward to reading a masterpiece from Peter Thiel. And I wish the next book from him is better directed to a specific audience in mind.

View all my reviews

Amazon RDS use case: Disaster recovery and scalability

In the previous post in this series, we talked of the of native database usage versus Amazon RDS. We established some reasons as to why you could choose to go for Amazon RDS. In this post we will see what Amazon provides in terms of disaster recovery(DR) and scalability. Amazon RDS makes it so easy to set up both of these that you just click on a couple of things here and there and you are done. Lets look at each of these individually.

Disaster recovery is when you lose your data in the event of a disaster and want to have an option to fall back on. To achieve this you keep a copy of the data you have, being updated in real time along with the original copy. I recently stumbled across a senior project management in a conference who told me:

Chances of losing data in cloud is almost zero. We don't need disaster recovery at all. We have to do it only because our customers want to hear we have DR in place.

Living in the information age and a background in database replication that was hard for me to digest. Lets bank on the WhatsApp acquisition of Facebook. There are multiple interpretations of why Facebook paid the whooping amount and to me the most compelling one is the user base (aka the logical asset) of WhatsApp as a platform. That is how critical data can be. In the age of personalization, data in my opinion is soon going to be the single most precious thing to own. Needless to say being paranoid about your data is the way to go. Make sure your data is safe at any cost and that is where disaster recovery kicks in.

Getting back to the start up use case, disaster recovery is something that needs careful implementation. Automatic failover is ideal but there aren't many database providers offering automatic failover with zero downtime, and neither does Amazon offer zero downtime. Assuming you are a startup and your aim at the moment is disaster recovery, not necessarily with zero downtime, you can again fallback on Amazon's multi-AZ configuration.

Multi-AZ configuration is an architecture where you have a redundant copy of your data being replicated synchronously. When I say synchronously, do realize that in theory this should impact performance. For all practical purposes, the impact is negligible and I would expect this to be something you can simply ignore. Setting up multi-AZ configuration is super quick with Amazon RDS. A couple of clicks on their UI and you are done. A word of caution again from the start-up perspective is that multi-AZ is twice as costly as a single server deployment. Thats the only flip side of having a multi-AZ deployment as I see it.

Lets now turn to the second subtopic- scalability. The most obvious solution to scalability is adding a server (in the technical lingo, slave) that replicates asynchronously from your main server (master). In my experience, I generally see people worrying too much about asynchronous nature of replication fearing data inconsistency. Think hard about your use cases- asynchronous could just be enough. And if it is, Amazon RDS has again something called read-replicas to cater to this. So you will have an architecture like this:

The source here is the multi-AZ deployment that we discussed earlier and the sinks are the slave servers. Once you have this set-up you can scale on the go. Whenever you see the bottleneck on database end, buy one more Amazon RDS instance and attach it as a read replica. More the scalability needed, more the number of slaves- simple enough! Of course this is not the ultimate solution forever because managing tons of slaves isn't the easiest job out there. But as a startup if you have hit this problem you are really successful and should now be able to afford some DBAs to advice you anyway. We will look at the problems on this end and solutions to them in a separate post. For now its time to look back and pat yourself on the back for being quite successful as a start-up- successful enough to have so much data. Keep your data safe and have fun with it!

Book Review: Direct From Dell

Direct From Dell: Strategies That Revolutionized an Industry by Michael Dell
My rating: 4 of 5 stars

This is an excellent book and I would recommend this to everyone who is starting a B2C company or holds (or aims to hold) a managing positions in such a space.

Michael Dell opens up about his life, how he started dell, what the differentiator for Dell as a company is and his obsession for the customer satisfaction. The book is divided into two parts: the first talks about dell as a younger company and the second focuses on managing the big organizers so you get both the perspectives. The examples quoted in the book, in the form of the experiences of Dell as a company shows how important a differentiating factor for a growing company is and why Dell prides itself on the support and customer feedback loop. The author talks at length about information versus inventory and highlights how important it is to hold as minimum inventory as possible. Alternatively focus on information flow. There is tons to learn from this book.

I am going to place this book in my "read again" section.

The reason I dropped a star (4/5) is that towards the end it fails to bind the reader. I got bored coming across customer obsession too many times and dropped the book just before it ended.

View all my reviews

Unpluggd Conference: Startup Launch and Demo session

In the last post we went through the overall impressions from the conference. Here is a summary of the first session of the day Startup Launch and Demo. Unpluggd invites applications for this slot allowing people to convey their startup ideas. Tons of applications follow and they choose the best of them to allow showcasing their products on the big stage. Lets go through the list and have a quick introduction to them.

1. Oyelife: aims to tap into the weekend leisure market. Their vision can be best portrayed by one of the slides that put up saying - "We think when people have time at hand in a weekend and don't know what to do they will turn to one of these three apps: zomato (for food discovery), bookmyshow (to book tickets for movies/plays/sports etc) and Oyelife."
   
   Bold statement and a courageous dream to pursue. Their content is curated and contains all the activities you can think of apart from the other two domains mentioned. I wish them well because I really hate hitting a dozen of apps. I will keep a close eye on Oyelife to allow me some less trouble on weekends to choose a way to have fun!
   
   Tapping into a bit of history, Oyelife comes from the same team that established Must See India which offers holiday packages. They claim to provide holiday packages with their own content (not user generated) like Housing does in the real estate ecosystem. They also claim to allow people to customize the holiday packages and provide 24*7 support. I am not very excited about this though I like the bold move in Oyelife.
2. Magictiger: is the one stop shop to getting everything online on your fingertips. At present they offer a phone number and you can contact them on whatsapp chat. They claim to cover anything you could ask them to get for you without any extra charge. Look at this screenshot from their website for an example:
   
   
   
   
   
   How cool is that? From booking a movie ticket to hotel, from ordering a food item to groceries they simply cover everything. Too good to be true and more technically too difficult to scale. Of course they were asked how they were going to be able to manage with one whatsapp number. And they said they have somehow been able to multiplex it and there are actually 20 odd people responding. There was a guy in the audience who actually ordered a nimbooz and got it delivered. Well this feels like a dream. Get lazy and have fun as it lasts and I really do wish it lasts longer!
3. MobiComKit: is a plug and play support for messaging. A lot of applications, though they can do really better with a communication interface embedded in them, end up either not having it to begin with or use a third party tool (like ola pays for the messages instead of embedding it in their app). If you don't have a way for people to come and spend some time on your website and talk about you, I think you are losing customers in the process. Its one thing to bring someone to your website and its a big deal to keep them there. And there is no better way to promote than a human interaction. If you use a third party tool, you lose data which I believe is the most important tool in your basket for people love personalization in today's age. Blame it on facebook or google, people have come to get used to being treated based on their behavior.
   
   MobiComKit adds an in-app messaging feature to your apps without the need to invest time on it without caring about the implementation or the infrastructure. Plug it, play it and engage your customers on your website.
4. Banihal is yet another attempt at match-making. Not again sigh! That was my initial reaction until I cam across Mr cheerful and confident founder. This guy had the best presentation skills. And he tried his best to convince that this is not just another match-making website. They are here to rule the industry. They have an impressive team and they already have a seed funding in too!
   
   Aaaaaand the investor is none other than David Cheriton himself. Here are the two gentleman:
   
   
   
   
   The guy who saw the potential in Google in-spite of many other search engines existing at that time and wrote the first cheque for them. Of course there were parallels drawn with this. Banihal claims to do it better and rule the space it is in. They are going to use neurology and all the research behind human decision making to make sure marriage is NOT just a decision. As Warren Buffett says- this is the most important investment one does in life. Banihal's vision is to give people 5 recommendations and they want to ensure the person does get married to one of them. Like Google, don't scroll to the second page of search results :) As much as I was uninspired by the space in which they are operating I really wish they do succeed for they have a good team, a visionary mentor and a confidence to rule the space- pretty much all ingredients to make a good business.
5. Chaska: to me came across a wannabe cool company. Because the name is catchy I was really looking forward to this. My immediate thought was it would be about food, but I  was wrong. Its a way to discover, re-mix and share videos. From the presenter to concept to problem to solution, I tried my best but I was never inspired. Because they are a startup I wish them well too but I would be surprised to see this make it big. I wish though that I am proved wrong.
6. uninstall.io: I like the concept though not a lot of other people shared my views. I have never thought so much about the % uninstalls against number of installs. I have never thought of analyzing this. Perhaps because I have never made an app myself? Maybe but to me this came up as something I would love to use as a startup. I would love to know why someone uninstalled my app, why someone finds my app crappy and how I can resurrect the situation. Go ahead and make it big guys. I do have faith in your ideas.
7. TrueSemantic: are in the space of customer support. Their vision is to improve the customer support and I wish they really do it in India for customer support most of the times in my part of the world is crappy. They came up with a statistic saying 80% companies think they have a really good and superior customer support while there are only 8% users who agree. Though I am no believer in a survey without knowing the source I would like to believe they are right. I would really expect a good survey to show that disparity. They have a really good website so I need not put much here. Just spend a couple of minutes on their well organized website and you will know what it is all about.
8. Twist: is an improved android gaming experience on the TV screen. If you have read my last post you would know I was excited about this. Now I am no gaming enthusiast for I have always loved outdoor games more but I think this is something gaming ninjas should surely try. Go give it a try! By the way, I am a big believer in a good domain name and I wish their website was just twist.com instead of the http://www.gettwist.tv/. But it depends on how deep your pocket is.
9. Audoce is one product I really want to lay my hands on as soon as possible. Imagine not having to carry keys everyday, forget a bunch of them. Imagine not having to come back early from office because your roommate lost the keys and imagine having it as a software with much more functionality than a conventional key and lock system. Your smartphone can tell you who entered the house and at what time, how long they stayed- this is just plain logging. Now imagine some intelligence (AI) built into it. Okay lets stop dreaming and support them to come up with a system that can help us trust this software for software is definitely more prone to being hacked than the physical keys. Go ahead guys and make a kickass product. There is a bigass market waiting for you to do well.
10. Drip is something that the NextBigWhat team itself has come up with. It offers short curated news stories and aims at people who just have 5 minutes waiting for their bus to show up to quickly browse through the news. This is an obvious market in the news publishing industry for there are so many news channels providing the same news over and over. Then there are aggerators like google news too. I am a bit confused about this project at the moment but given their contacts I would like to believe it is easier for them to do a good job. All the best to the NextBigWhat team and kudos for the great work on the conference side! 

I will keep a close eye on this list and I would love to know your views on these wonderful ideas.

Unpluggd Conference: Experiences

NextBigWhat is a meetup to gather together geeks, entrepreneurs (aspiring ones too!), angels and investors.  I am pretty sure that's not an exhaustive list but in general its a conference centred around tech startups aiming to get all the concerned parties together on one platform. You find IBMs and Microsofts shouting that they are all for helping startups grow and find their feet, you find some kickass stalls showing demos of their first product, you come across their business counterparts trying hard to just kick-start a conversation with you, you even come across investors looking for people ready to pitch their ideas in their fields of interest and trust me I did find one saying on stage- if there is someone who execute a certain idea he was sitting ready to fund it. Now you can't have a better invitation in your life!

Moving on to the conference, it was a two day event- productgeeks and unpluggd being the theme of the two days respectively. I regret missing the first day because it was a Friday and I really didn't have a lot of leaves left. That brings me to the dilemma:

Should two day conferences be held on Friday and Saturday or Saturday and Sunday?

I will be happy to know where you stand there but for now lets stick to the topic at hand today. As I didnt attend the productgeeks conference I will only leave you with the agenda and speakers frm that part. I will make a point to break into this next time.

Moving to the second day- Unpluggd! I reached the reception and found a identity card for myself. I had pre-booked for the conference. You could also book your ticket on the spot. I grabbed my ticket and moved ahead with five minutes in hand to have a quick look at the stalls. The first to grab my eyeballs was a product called Twist. In a nutshell Twist provides the same fun that gets you hooked to your android phone on the bigger screen of your TV. I did play Temple Run and it was real fun! Instead of the remote that you would generally use you can use your smartphone to play on your TV. Doesn't that sound cool? They also promise to provide innovative controls and everything at a lower cost compared to their competitors. Well I am not sure If I can buy into that but there is only one way to know that- to try it! 

Next, I hopped around to find http://www.businesssetup.in/. I grabbed their business card and was trying to escape quickly when a girl requested me to share the idea and signup etc. I signed up wit them and quickly browsed across the other stalls to end up in the auditorium in time. The presentations were to begin at 9:30, the first being "Building Freecharge: Lessons learned" by the Freecharge (now part of Snapdeal) founder and ex-CEO Kunal Shah . Kunal could unfortunately not be there in time and hence the day started with the second item on the list- Startup Launch and Demo.

This session consisted of 10 startups. All of them in their infancy presenting an account of themselves. They came up with the demos, presentations, stories etc. The next post in this series introduces each one of them.  We then had a networking session over tea and breakfast which lasted half an hour. The following session was possibly the most high profile one. There was Rahul Yadav (founer & CEO Housing.com) being interviewed by Akash from the NextBigWhat team. Rahul was greeted very cheerfully by the crowd. But contrary to the celebrity status the news publishing media has attached to him, he came across as someone who is so down to earth. Rahul, though he looks like the guy next door and has no qualms about his tremendous success, came across as an inspiring person to me. He just seems to know what he wants and is amazingly clear in his responses. He came across, to me, as someone, who is very intelligent, querky and frank, yet very sorted. There was zero percent pretension, personality and all. After the interview, the floor was open for the audience to ask questions to him directly and there was an ocean of questions awaiting him. There were so many people just wanting to talk to him somehow. A couple of striking takeaways from this session from me were again clarity of thought, innovation vs problem solving (the obvious problems too), the importance of execution, his unique way of building the team etc. 

Next was a small talk by Nilkanth Iyer, Country leader cloud ecosystem India- South Asia and he went on to make a point about IBM's commitment to supporting startups. Next there was some good food waiting for us. It was a nice one hour session allowing people to network with whoever they wanted to. Following this was a session from Yes Bank COO. This session like it was scheduled after the lunch break was quite sleep inducing to me. Ok let me admit I did manage a 5-7 minute sleep in this period actually. But as far as I understood (very little though!) he seemed to claim that Yes Bank is different, it believes in eliminating problems and not only help as a banker. Navigating through the problems, the entrepreneurs or any customer Yes Bank feels to arrive at the root cause and eliminating it, their support for the startup ecosystem in India etc. 

Next was one of my favorite sessions- a panel discussion titled "Is there an appetite for More FoodTech Funding in India". The guests in the panel were Sanjay Swami (from AngelPrime), Mast Kalandar CEO Gaurav Jain and the SpoonJoy co-founder Manish Jethani. The moderator started the discussion essaying that we are a very populous country and we love our food! We have 3-4 meals per day to say the least and thence this clearly is a big market, yet still untapped leaving a lot of scope for the aspiring entrepreneurs. Gaurav and Manish portrayed their stories and went on to list the problems they faced and lessons learnt. Sanjay encouraged the audience to explore this field and even went to the extent to say that if someone can show him a way to subscribe so they could completely take care of his diets, taste and variety etc at the same time promising him a weight loss of 5-10 kg he was absolutely ready to fund it. The discussions went on to essay the difficulties Gaurav and Manish faced to what they think are some things one should take care of when trying out a startup in this field.

Following the next and last tea session, there were the sessions I was mostly looking forward to involving Abhay Singh (from InMobi), Aparmeya (the guy behind TaxiforSure) and Kunal (freecharge co-founder). Very insightful, frank and masterful advises. Take this quote for an example from Kunal - 

Make a kickass product with a smartass team for a bigass market!

More insights from this session in the next blog post. See you there!

The Amazon RDS use case: start up point of view

Coming from three years of experience working as a developer on MySQL, I always felt Amazon RDS (Relational Database System) was just snapping away customers from the database companies themselves by only making some wrappers around it to offer RDS. I hated the idea of using Amazon RDS instead of a database. In the last one month I have read a lot about Amazon RDS and my views have changed. I have started to understand that the amazon RDS does have a real good utility at times and does a god job too. Below I talk of the clear market I see for amazon RDS. Note that all content below is written from the point of view of a start up or a small company which is where I see Amazon RDS fitting the best.

I am a firm believer in the popular quote- Life is a sum total of the choices you make. When it comes to technology today, there are so many choices that the toughest part is choosing your software correctly to suit your use cases. One such problem I recently came across was maintaining a native database (backed by MySQL, postgreSQL or Oracle) or choosing to go with the Amazon RDS. As with all difficult choices it actually depends on the state of organization the most dominating factors being:

1. The amount of money you are willing to spend. If you are broke on this front affording Amazon RDS is out of context.
2. Flexibility- While the Amazon RDS does a great job of making things simple, in my opinion, it happens at the cost of losing flexibility that comes with using a native database. A counter question to ask yourself is- do you really need so much flexibility at this instant?
3. In-house expertise: If you have guys carrying a real good experience on of the native databases, it is a good indicator to take on the challenge and go with a native database set up. Be assured though that shit happens and I would imagine more so with the native database in a five people company.
4. Available time- small companies live and die by how much work they can get done on the application side. If you are a small company racing against time to add features to your product, this should seal the deal and you should should decide to use Amazon RDS even though it costs a little bit higher.
5. Lastly, it all depends on your product or application and the nature of data it carries.

In a nutshell, the choice narrows down to the time you are willing to afford to work on the database end . Lets first see what tilts the argument in favor of Amazon RDS. The big thing that Amazon RDS enables is to save on labor for all sorts of things, for instance:

1. Monitoring,
2. Logging,
3. Auto-scaling,
4. Caching infrastructure,
5. DB servers,
6. Media transcoding etc.

So Amazon RDS frees you from thinking about all of that by paying some extra money and biting on your flexibility as explained in the introduction section. For the extra cash you burn you save on the labor involved in doing all of them.

If you choose to set up your own database and manage it, be it on your dedicated hardware on an Amazon EC2 instance, you need to realize that you will have to periodically invest in it

1. To thoroughly document the set-up,
2. Very thoroughly test the replication, failover behaviours, backup/restore flows etc.
3. You will need to build a team to do most or all of this.
4. Upgrade (whenever required) yourself
5. You will have to go through the practice runs (backup, restore, failover) all over again to make sure everything still works.
6. Remain failsafe so if postgres fails in a weird way, you need to carefully punt parts of the infrastructure in a pre-determined way to avoid data loss etc.

As a rule of thumb if running your own native database would markedly improve the product, do it. If not, you are probably looking at trading upfront cost for labor expense. For a small company with low manpower, I think one should really let Amazon RDS handle everything and focus on the product for the time being

Sadly, this is an excerpt from the notes I had while studying this subject. At that time there was no idea of writing a blog and hence I have lost the references but all this is mostly derived from quora, stackoverflow and the official manual for Amazon RDS. Lastly, best of luck on managing your data. Keep your data safe and have fun!

The story of database scaling at YouTube and Dropbox

I have recently been studying postgreSQL. Coming from the MySQL Replication background and studying postgres, it was natural to just flow into their replication module and I couldn't help compare the replication modules of two of the biggest open source databases. I have also been studying ways to scale a growing database and trying to figure which of these two databases is better at scaling- in terms of ease of use, stability and performance. I have always been interested in scale and distributed systems and I am a big time follower of the architectures of the biggest web companies.

As evident from my last few posts, I have also been reading a lot about entrepreneurship lately. Combining all of this, when I come across a resource that tells me how a YouTube or Dropbox grew in scale, what are the problems they faced while their user-base grew and how they overcame it, gives me immense joy. Recently I came across two videos that I want to share.

The first is the story of YouTube presented pretty well by one of their earliest developers. The video clearly depicts the technical fine prints, the desperation to keep their growing site live and the common problems scale tends to bring up. There are times when you think something is so well presented that you don't want to reproduce it in your own words and thence I will just leave you with this video. Have a look at the amazing presentation.

The second video that I want to share today is the story of Dropbox. The speaker begins with a simplest architecture the founders started with and goes on iteratively to show how they kept improving their back-end. Note that Dropbox is a write intensive workload and hence this is a real fun video. Have a look.

Lastly, if you have more such references, do share along.

Book Review: Young Turks

Young Turks: Inspiring Stories of Tech Entrepreneurs by Shereen Bhan
My rating: 3 of 5 stars

The unique selling proposition (USP) of this book is that you get introduced to the founders of flipkart, snapdeal, justdial, druva and more. The book is for someone who knows nothing about these guys. Having read about entrepreneurs before I didn't get a lot of new information.

Secondly, you can see through the book that this is a journalist asking questions. The books doesn't give a detailed account of the founders, nor is it inspiring enough. Shereen Bhan seems to have a predefined set of questions she asks everyone. This starts to get boring beyond a certain point. The stories are not personalized enough. I have read some of Rashmi Bansal books and she does that quite well. In the current book, the stories are quite short and I found the author moving to questionnaires too quickly for me.

Having said that this is a good starting point for reading books on this topic. Go for it if you do not know a lot about these founders already. If you have read a fair bit on this topic, skip this.

View all my reviews

Book Review: How I Braved Anu Aunty & Co-Founded A Million Dollar Company

How I Braved Anu Aunty & Co-Founded A Million Dollar Company by Varun Agarwal
My rating: 4 of 5 stars

The story of Varun Agarwal is quite inspiring. The best part of this book is that you can relate to the central idea. We have all had an Anu Aunty in our lives and though you can see that the story is tweaked at times to make the presentation good, this is something that the author openly accepts in the book. Fare enough, the presentation needs to bind the readers.

Secondly, as a Bangalorean there are tons of places you can relate to and its fun to come across those references. All in all, the book makes some really good points keeping the mood light, cracking jokes along the way. This is one book that got me interested in reading books once again. I would recommend this book to anyone who is in the struggling phase, anyone who has lived in Bangalore and anyone who just wants to read a book for fun. Go for it, its worth a read!

View all my reviews

Book Review: Life is What You Make It

Life is What You Make It by Preeti Shenoy
My rating: 2 of 5 stars

A good measure of whether you like a book or not, to me, is the feeling you carry soon after finishing the book. And with this book it was NOT a positive feeling for me. The book seems to over-dramatize certain instances.

I read this book between two masterpieces by George Orwell-
Animal Farm and Nineteen Eighty-Four. Perhaps that influenced my opinion on this book too and I can't help compare these books. The current book stands nowhere if you compare with those standards. You can clearly see the difference between the two authors. While on one hand George Orwell paints some beautiful pictures and maintains the tempo throughout the book, Preeti Shenoy fails miserably there in my opinion. There are parts in this book where you just don't feel like continuing. I would't recommend reading this book. There are tons of options out there and though I haven't read a lot of books myself I am sure there are plenty of better books read than this. This book can kill your interest in reading and I almost fell in that trap.

View all my reviews

Book Review: The Start-up of You

The Start-up of You: Adapt to the Future, Invest in Yourself, and Transform Your Career by Reid Hoffman
My rating: 4 of 5 stars

I will divide my review for this book into two parts. One to project what the author conveys and two- to show how this book is different from others.

In our part of the world, we are starting to move from the get a job mentality to start a company and thankfully so. The way we look at the entrepreneurs and people employed by the bigger organizations is completely different. Obviously, not everyone can start a company. Then does it mean that the entrepreneur mentality should be borne by only those few people starting a company? This is where this book fits in.

The author goes on to essay that there is no clear line of distinction between the two sides. The author suggests that entrepreneurship is something pretty central to whatever one does. The author shows how the concepts of entrepreneurship apply to regular jobs.

Moving to the second point of how this book is different- the first thing that I try to find when I come across a book is if it will impact my thinking in some way. Frankly, not many do that. This book, if you follow the exercises definitely does that. Go ahead, read the chapters one by one and follow the exercises. You will see the point behind linkedin and other social networks. You will know how to use first, second and other degrees of connection. Your network will grow and you will learn to use your network in the right way and much more. A must read for everyone!

View all my reviews

Book Review: Connect The Dots

Connect The Dots by Rashmi Bansal
My rating: 4 of 5 stars

I had been reading a lot on entrepreneurs when I was recommended this book. But before this book, it had only been about massively successful founders like Elon Musk, Steve Jobs, Bill gates etc. Reading only about them gets you into thinking big but at the same time you feel a bit intimidated by these personalities. You keep asking yourself if they were born geniuses, if you too could do something etc.

Reading this book opened my horizons. The book covers some lesser known entrepreneurs and goes on to show that its not only limited to software, its not only limited to silicon valley but entrepreneurship is all about seeing things around you and starting whatever you feel for. Its more basic than choosing a field you want to be in and finding a problem there. Its more about finding an opportunity in any field, any scale and any place.

All in all a great piece encompassing a lot of variety. Indeed like the title says it did connect some dots for me. If you are dreaming and not able to find the one thing you would want to do, go for this book. If nothing else it will at least broaden your range so the problem is a touch simpler.

View all my reviews

Book Review: Screw It, Let's Do It

Screw It, Let's Do It: Lessons In Life by Richard Branson


Richard Branson opens up about his life quoting instances to make the points listed in the contents. A short one and a good read especially if you are into reading about entrepreneurs. Richard's story has tons of things that one can learn from.

I read this book after the other one - Losing my virginity and it felt like this is just a shorter version of that book.

If you want to read more into the instances he quotes in Screw it lets do it I would recommend reading the other book. On the other hand, for the same reason I wouldn't recommend reading Screw it lets do it if you have already read the other book.

View all my reviews

MySQL 5.7.6: It is easier to switch master now!

Introduction

One of the primary objectives of MySQL replication is providing an easy failover process i.e. switching to a redundant system if the primary MySQL server fails. In MySQL this translates to switching to the most appropriate slave server in the eventuality of a failure of the master server.

The promotion of a candidate slave to become a new master is based on a lot of factors, undoubtedly the most important factor being that the chosen slave should be most up to date with the primary server (the old master) before we lost it! This blog explains how to use new features in MySQL 5.7.4 and later to make failover easier.

To find the most up to date slave server, a failover script looks at the set of transactions received by the slave and compares it with every other slave to find the one that has received the biggest set of transactions. There could be more sophisticated ways of doing this, for instance you could choose one preferred slave that you want to promote (perhaps it has a better hardware configuration, it has no filters, physical location etc) and make sure it receives every transaction that has been received by all the other slaves. For simplicity though, let’s narrow down our definition of the most appropriate slave to promote to be the one that is most up to date with the lost master.

How to failover using GTID based replication

To denote a set of transactions, MySQL uses GTID sets (a set of global transaction identifiers). To read more about GTIDs, you can refer to our official documentation or developer blogs. To find the set of transactions received by a MySQL slave server, you simply execute:

mysql> SELECT RECEIVED_TRANSACTION_SET FROM peformance_schema.replication_connection_status;
+------------------------------------------+
| RECEIVED_TRANSACTION_SET                 |
+------------------------------------------+
| 4D8B564F-03F4-4975-856A-0E65C3105328:1-4 |
+------------------------------------------+

Execute this on every slave and compare the sets to find the slave with largest received transaction set- that’s your candidate slave for promotion. Let us call this slave the new master. Before you switch a slave to replicate from the new master, you earlier had to make sure all the transactions the slave has received are executed. In versions of MySQL prior to 5.7.4, you needed to do the following steps:

1. stop slave.
2. start slave to replicate until all received transactions are executed.
3. wait until all received transactions are executed.
4. switch master to redirect slaves to replicate from new master.
5. start slave

This would translate to the following MySQL commands:

1. STOP SLAVE;
2. START SLAVE UNTIL SQL_AFTER_GTIDS= <received_transaction_set>;
3. SELECT WAIT_FOR_EXECUTED_GTID_SET(<received_transaction_set>);
4. CHANGE MASTER TO <new_master_def>;
5. START SLAVE;

However, in MySQL-5.7.4, we introduced a feature which allows one to selectively stop only that component of replication which requires a change. This means that in the present context, to switch to a new master we only need to stop the receiver module (or in technical terms the I/O thread). The applier threads can continue applying transactions, if there are any pending, while we switch master. Building on this infrastructure we can now reduce the above steps to the following:

1. Stop the receiver module (STOP SLAVE IO_THREAD).
2. Switch master (CHANGE MASTER TO <new_master_def>).
3. Start the receiver module (START SLAVE IO_THREAD).

Note the removal of the wait function (wait_for_gtid_executed_set) to ensure that the received transactions are executed before you switch master. There is no need for that step anymore!

How to failover using non-GTID based replication

If you are not using GTIDs, though we highly recommend you should, you can still take advantage of the current improvements. This means you can change the following part of your failover script:

1. STOP SLAVE
2. SHOW SLAVE STATUS to get coordinates (Read_Master_Log_Pos, Master_Log_File)
3. START SLAVE UNTIL coordinates saved in step (2).
4. SELECT MASTER_POS_WAIT (coordinates,...).
5. CHANGE MASTER TO <new_master_def>.
6. START SLAVE.

to the following simpler steps:

1. STOP SLAVE IO_THREAD;
2. CHANGE MASTER TO <new_master_def>;
3. START SLAVE IO_THREAD.

Conclusion

Using the improvements in the newer versions of MySQL 5.7, failover becomes easier. Our effort to improve MySQL high availability continues and we remain committed to easing the processes. Please do try this, and as always let us know your feedback. You can also use mysqlfailover or MySQL Fabric that automate the failover process.

MySQL-5.7.6: Introducing Multi-source replication

On March 10, 2015, we released MySQL-5.7.6 and among many other things it includes multi-source replication which provides the ability for a MySQL slave server to replicate from more than one MySQL master. We have been very careful with multi-source replication in terms of what exactly our users want and we have tried our best to incorporate as much feedback as possible. In the process, we released the feature twice under MySQL Labs asking users to try it out and tell us:

1. If this caters to all their use cases,
2. Plays well with the existing applications using single source replication so our users don’t have to change their old scripts at all if they do not intend to use multi-source and instead stick to single source replication.
3. The user interface is in sync with our naming conventions to date, easy to understand and intuitive etc.

Note that the look and feel changed as we moved from one lab release to another and finally to the latest version as MySQL-5.7.6. In this post, I aim to introduce the released feature, the commands and how you can monitor the multi-source replication internals. Let's start with the following figure that best illustrates the core of multi-source replication.

In the figure above we have three MySQL sources (independent MySQL servers- master 1, master 2 and master 3) replicating to the only slave acting as a sink to collect all the data from all the three sources.

The use cases of multi-source, as you have probably already guessed, are related to data aggregation. Note that there is no conflict detection or resolution built into multi-source replication. We expect the application to make sure that data coming from different sources are non-conflicting. A typical setup could be something like this:

The same concept could be extended to shards (instead of databases). So another use case of multi-source replication is to join shards and make a full table on the sink (aka slave). .

To understand how to configure and monitor multi-source replication, we introduced the notion of channels. A channel is an abstraction of the internals of MySQL replication's finer details. It hides the machinery underneath while providing the level of detail that helps the user manage and understand multi-source replication. From a user perspective you can imagine a channel as a pipe between a master and a slave. If there are multiple masters, there are the same number of channels (or pipes) emerging out of the slave server as the number of sources as shown in the picture below:

If you understand MySQL internals already and want to know exactly what constitutes a channel, look at the pink strip in the following picture. The replication channel documentation has all the details. But if you don’t know these details already, ignore this figure and move ahead. After all that is what we wanted to achieve with the concept of a channel.

With the concept of channels established, you can now follow steps described in the tutorial section of our official documentation to work with multi-source replication. Note how the FOR CHANNEL <channel_name> clause now allows you to take each master-slave instance individually and work with them as if you were working with a single source replication topology.

Having set up multi-source replication and making sure there are no conflicts you can expect it to just work out of the box. But if you want more details you could look at our monitoring interfaces to provide you the details on every channel. In MySQL-5.7.2, we introduced performance_schema tables to monitor replication, the good news is that these tables were always designed with multi-source replication in mind so they should work seamlessly with multi-source replication. All six replication performance schema tables now have a “channel_name” field added to them to individually access the configuration and status on each channel. As an example, we have described performance_schema.replication_connection status in our manual. Given that you are working with multiple channels, lets walk through this table again and see how it presents the internals. Try out the following query to look at the receiver module:

We can see quite a few things here:

1. Which master does channel1 replicate from?
   The one whose UUID is as given in the “source_uuid” column.
2. Which thread is responsible for receiving transaction through channel1?
   Thread number 13. Note that this thread number is same as the one in performance_schema.threads table. You can use this information to now look into other performance_schema tables to mine more statistics using the joins.
3. Is the receiver module of channel1 active or down at the moment?
   The channel is active and receiving transactions because its service state is ON.
4. What transactions have been received via channel1?
   Transactions 1-4. See the global transaction identifiers in the field "received_transaction_set".
5. What if there was an error?
   The last three fields in each row give an idea of the error on that channel, if any.
   See  the example below where channel1 has lost connection:
   
   
   
   
   
6. How about my network?
   Look at the last two columns to get an idea of this. The last time a heartbeat signal was sent is shown in the “last_heartbeat_timestamp” column.
   The "count_received_heartbeat" indicates how frequently heartbeats are being sent out. A big number here would mean that either the connection is not stable or this channel was idle having nothing to replicate.

Likewise there are more replication performance_schema tables that you can use to get to finer details. If there is anything else that you think should be available in these tables per channel, leave us a comment and we would be happy to take it forward. We truly believe in working with our users and look forward to your experience with this feature. There is a lot more to explore in multi-source replication so go try out the feature and if there is anything that you have suggestions for, please do leave a comment. Don't forget that MySQL 5.7.6 is a development milestone release (DMR), and therefore not yet declared generally available.