Fraser Xu Thoughts on life and code.

2016 Review

先来回顾一下15年定的小目标

一直都没有对下一年有过什么认真的规划,因为觉得自己的新一年充满着无数的变数,根本规划不来。但是可以设定一些小小的目标,即使不完成也没有关系。

那个时候才刚过来墨尔本一个月,不知道这边的生活会是怎样,也有很多新东西需要去了解和接受,所以不知道要定什么目标。现在回过头来看也对,这一年的生活确实是没有办法预料的。

有机会的话就出去走走,没有的话也没关系。

16年相比15年走的地方少了很多。大部分时间留在墨尔本,期间借着参加CampJS和JSCONF China去了次悉尼和南京。自己本是计划可能会去新西兰或者欧洲或者美国但是都没有实现,但是没去成也还真买啥关系。和这大半年的工作有很大关系吧,后面再慢慢解释。

在国内的时候就不太喜欢人多的地方,也不喜欢在人多的时候主动去找人交流。

过来这边算是找对了地方?澳洲的各个地方对比中国,全是地广人稀,去哪都没有什么人。加上自己一般不主动去人多的地方,所以单从社交这一点几乎是没有什么变化的。

16年主要干了些什么

来澳洲最主要的原因就是工作,所以来的第一年,生活差不多都是围绕工作展开的。

来公司后加入的是公司目前盈利最多也是最大的一个项目Envato Marketplace,听起来好像是加入了最好的团队,但从技术角度来讲并不是。这个项目的代码让我第一次听说了什么叫做Monolith。我自己之前的经历主要是JavaScript开发,前端和后端都写点,熟悉Node.js的开发人员应该都知道这个社区提倡模块式开发,只要是能从核心业务中单独分离出来成为一个模块的我们都会尽量把它写做一个模块发布到NPM然后到项目中使用。但是现在这个项目(主要为Ruby On Rails)因为历史技术和产品等各方面的原因,把下面7个子站点的大部分代码放在了一个仓库,这样的结果是随着时间和产品复杂度的增加,代码会变得更加难以上手和维护。当然Monolith和Modular Development各有各的好处,我这里也不做过多探讨。

提到这个是因为我是以前端工程师的身份加入到这个项目中的。传统的服务器端Ruby On Rails渲染页面,前端使用jQuery, CoffeeScript, Bower。因为项目起步比较早而且大,用的Rails的版本也比较低,前端的工作和后端的工作相互依赖,就导致了依赖版本升级困难,开发进度受到各自的影响。

另外个人过来之后觉得最大的问题是自己的开发效率也因此受到很大影响。第一次需要clone好几个G的代码,安装各种依赖,数据库同步,启动各种服务。修改一行CSS代码在浏览器刷新看到效果需要10s以上的时间。在这样的环境下工作了大概一个多月,做的实在不是很开心。

公司其实内部也有其他新的项目和团队,因为没有历史包袱,都在技术方面敢于尝试使用新的技术。就前端方面也比较能够跟上业内的节奏。可能因为自己之前在公司做过几次前端方面的分享,其他团队的Manager好像知道我之前的技术背景和他们现在的符合,加上他们有前端近期要离职,所以找我谈了问我有没有兴趣到他们团队去帮忙。

那个时候的我当然是想去的。于是主动约了自己所在团队的Manager的Manager, 表明自己的立场,在那样的前端环境下觉得自己没有进步,工作效率不高所以做的不开心。楼下团队找我下去帮忙,问让不让走。

其实在约谈之前,自己知道,公司招人的时候都是按团队招的,一般来讲哪个团队招的人在哪个团队工作,内部换的不太好。所以自己的想法是如果现在的团队能够让我在现在的项目中用到新的技术,改善前端的技术和开发体验,我还是会优先留下。如果团队实在没有办法接受这个建议,那么我就换到其他团队去。

另外一点自己比较有把握的是,公司的开发团队不管是管理层还是普通开发人员,很多也都了解现在的状况,都有尝试作出改变的想法,但是却没有好的机会去具体执行。还有就是这里的Manager还是比较能听进去建议的,所以在谈话之前我就能够猜到大致的结局。

正是那次谈话,决定了我在那之后的工作内容和方式。先是我一个人脱离其他团队,从项目中的一个非常简单,且访问量一般的页面开始,花了一周的时间完成了一个简单的Node.js + React.js服务器端渲染的应用(期间也尝试过用Graphql但是没有成功).

在那之后”碰巧”遇到公司内部结构调整,我们分出了一个新的Foundation团队。这个团队和其他团队不一样的是没有具体的产品开发需求,相反主要去研发一个新的基础开发平台,让其他的技术团队能够用这个团队的成果更快更好的去完成他们的开发需求。自己也就很幸运的加入到了这个团队当中。

接下来的大半年差不多就是和新的团队一起围绕这个目标在工作。开发管理团队也是冒着很大的风险努力说服产品团队,所以也给我们团队定了很明确的预期时间目标。做的好大家都开心,做的不好队伍可能就要解散。

项目进行到目前还算顺利,按计划大概1月底到2月初能够用我们的平台在线上运行产品的核心搜索页面。搜索页面算是整个产品访问量最大的页面,根据统计访问量(统计一般只针对单个站点但是同样的代码和服务器实际运行了下面的7个子站点)应该也在全球前100,加上我们使用的技术团队的其他人也都没有足够多相关的经验,所以挑战不小。

因为到现在项目还没有正式上线,所以更多的技术细节可能之后再单独的写出来跟大家分享。

Revolution or evolution? Who knows.

一些做的不好的地方

  • 和亲戚,朋友,新老同事的交流变的更少了
  • 文章写的少了,技术的非技术的都是
  • 开源项目代码贡献和维护变的不够积极
  • 技术分享的频率在逐渐下降

一些有意思的事情

  • 在驾照还没考到之前买了一台二手🚗,而且无照驾驶还去了不少地方
  • 停用国内手机号,QQ密码和Weibo密码都忘记或被修改而且找不回来了
  • 以讲师和组织者的双重身份参加了JavaScript中国开发者大会,做了题为Learning design patterns from modern JavaScript frameworks 的分享

2017年可以有哪些改变

第一件事情是把驾照拿到,这个假期过的这么折腾是个教训。

第二件事就是写10篇以上的文章。

第三件事就是至少去到一次澳洲和中国以外的国家。

最后一件事就是努力将现在的项目继续在公司推进下去,在项目上线后能够运行稳定。花更多的时间提升项目性能,优化开发体验。同时推动团队的前端开源氛围。

Cheers.

My First Production Isomorphic React Graphql Project

The story

During the past few weeks, I’ve been given the opportunity to rebuild the front-end of a project with “modern approach” to replace an existing CoffeeScript, jQuery, Bower based app running on Ruby on Rails.

After about 2 sprints of work(2 weeks for each sprint), we shipped our first version to production last week.

Before I started to share my experience, I’d like to give an overview of the architecture for the project.

The current stack

  Library
View React
State management send-actions(like Redux, but simper)
Date fetching GraphQL, Relay
Route React-Router
Assets serving Webpack
Precompile JS Babel
Server Node.js(for server side rendering React)

Why the current stack?

I’ve worked on lots of different projects before with different stacks. And I always have the idea to not use any boilerplate in mind when start a new project. Boilerplates are usually built by and for people with different requirements for a project, and none of them are identical to the one you are trying to build. So usually I will only keep a list of well maintained boilerplate project, and only use them as a reference when my own stack gets into trouble.

The new project has a few requirements:

  • Server side rendering for progressive enhanced experience so the page could work for user without JavaScript
  • SEO, we are mainly an e-commercial website, so SEO is the number one priority
  • The app needs to talk to a couple of micro-services, and tokens are usually stored on the server for safety reasons
  • UI state should persist from url, not only for SEO, but also for a better user experience
  • Fast iteration time, to move fast and delivery better user experience
  • Improve performance, the short time we delivery page to user, the longer we can keep the user on the website

There are also other requirements which are not for business, and most of them are actually for a better developer experience.

  • Babel. For use a couple of handy syntax today that are only available in future browser
  • Webpack. For compiling assets, hot code load, uglify
  • Modern JavaScript libraries that have best practices in the community

With the above requirements, I started from the simplest hello world express server, and deployed it to Heroku. The other day I started to build the static part of the page, and since the code need to render from the server side, I installed React and render a few of Header and Footer component and rendered them on server with React.renderToString.

Since I also need to have other pages like 404, 500, I added React-Router to have the router support. It works super fine and I love what I did so far.

But I think I forget to mention the setup I need to make in order to make those 3 libraries to work for both browser and server. Here’s the dependencies list:

  • babel-cli
  • babel-core
  • babel-loader
  • babel-plugin-transform-runtime
  • babel-preset-es2015
  • babel-preset-react
  • babel-preset-stage-0
  • babel-register
  • react
  • react-dom
  • react-router
  • react-hot-loader
  • webpack
  • webpack-dev-server

And along with them I will need to have 3 webpack configs.

  • webpack.client.config.js
  • webpack.server.config.js
  • webpack.dev.config.js

And another .babelrc which include the babel plugins.

I’ve used Webpack in several projects, but it stills feels very hard to make it right each time I do it again. I’m using lots of plugins for different purposes in my configs, but to be honest I can’t say I know what exactly what those individual plugs does in the project. Some of them are inherits from Babel 5 and I may know what it does, but others may be something totally new only in Babel 6, I don’t have the time to go through each of them.

It’s actually an OK experience so far, I managed to have a nice working demo. And I could show my manager what I’ve achieved so far in a rather short time.

The next step is to be able to load dynamic content through our api-gateway. Since we only want to keep the token for internally use and don’t want to expose it in the browser, we had the idea to build a simple “proxy” server which re-direct the request from the browser and then pass the request together with token saved on the server to make a request to the api-gateway.

In addition, if a page need to load multiple results, we could make them together into one and have a custom api endpoint. For example if we want to get the stats total user and total items on the page, normally we will need to do two individual requests, but with the “proxy api” we could do this via a /api/stats so the browser only need to make one request. This could help user on a mobile device since network request on the server side is relatively reliable and faster.

When we get to this stage, we came to the idea of trying our GraphQL, because the “proxy” server is similar to what GraphQL wants to achieve, and in the future we will have more complex logic, the benefit of small pageload and flexible query language could help us in long term. Given that this is an experiment project so we decide to try it out.

To get started, we need to install the following packages, and it still kind of making sense to do so:

  • express-graphql
  • graphql
  • react-relay
  • graphql-relay
  • babel-relay-plugin

Oh, wait! Since we are using react-router, we also need a tool to fetch the data based on the current router so we could fetch all the data before rendering the page:

  • react-router-relay

And, last but not least. We are doing isomorphicuniversal app, so we still need to do something to make them work on server side, let’s install some more other plugins:

  • isomorphic-relay
  • isomorphic

Okay, I think we are almost there, we’ve got almost every tool we need. We’re going to ship the product to production.

With this setup, I got problems that does not belong to any of the framework itself, but how to make them work together. I’ve heard lots of success stories for isomorphic application, I’ve also heard people talking how awesome GraphQL and Relay is for data fetching. But I can hardly find any live example of using all of them together.

Here’s a few pain point I met while hooking them together:

  • process.env management for application running in different environment(dev, ci, staging, production), most boilerplate project do not cover that
  • process.envmanagement for isomorphic applications, using them with Webpack is tricky because it’s mostly design for client-side code, and convert variables from variables to strings and then back to variables to make them work for both env
  • debug with source map support for compiled code for different environment
  • the .babelrc file that varies on different environment

Luckily, with enough time spend on those problems and wonderful resources over the internet, I managed to make the whole stack working, and we shipped it to production last Friday.

After shipping it to production

Yes, please hold on and stop telling me you should never ship something to production on Friday. This is something we all know as developer, there’s a few assumptions that we’ve made when ship the code:

  • We have run the code on our staging and production for 1-2 weeks
  • we have complete monitoring services that show the metrics of the app
  • It’s a shiny new stack and we can’t wait until next week to ship it

And those assumptions are wrong.

All those metrics we have are not facing really user, it’s behind a domain hosted on Heroku app, and we’re the only user who knows about it. And once we resolved the DNS to use the new domain(which takes about 1 hour to take effect), we started to get some new metrics from our tracking services. It was all good at the beginning.

But not until a few hours later, we found that the memory usage of the server keeps going up. Even though I was almost sure there’s a memory leak somewhere in our code, but with this shiny new stack I had no idea what could go wrong as there’s so many possibilities. Normally you could just revert your code to a previous working commit and do a re-deploy, but we don’t even have one.

I ended up sitting in front of my laptop the whole Friday night to watch the memory usage goes up and I restarted again, and wait until it goes up to restarted it.

The other day with the help from some of my nodejs developer friends, I added a process management tool called pm2 which restart the server when memory goes to a max_memory_restart limit so I could have time to have a rest and time to figure out what’s going on.

But pm2 could not help fix the memory leak issue, so the rest of the week I started to look at nodejs profiling solutions and find out a few technicals to find potential memory leak. It’s not the topic of this post but all I can tell is that is hard. Especially you have such a setup with isomorphic babel compiled code base.

Have you find out the memory leak yet?

The answer is no. After discussed with the team, we decided to remove all GraphQL and Relay related code. And here’s the result:

memory usage

We all know that is totally unnecessary for the app, we were keeping it to prepare the code for the future.

Conclusion

So here’s a few lessons I learned from this project:

  • Don’t ship code to production on Friday, for whatever reason.
  • You can plan big, but always start small.
  • There’s no reason should stop you from trying new libraries, but chose the right timing.

Last but not least, I’m not a super fan of hearing people complaining about JavaScript fatigue, this post isn’t about complain them at all. It’s all my personal fault to use a library at a wrong time(or too many libraries at the same time). And it’s also the reason I explain step by step why I bring in those libraries into my project.

It is because whenever I get a problem, those tools, build by lovely open source developers who spend their spare time, helping us solve our problems for free, they are the one making our life easier. If it happens to solve your problem, always appreciate their hard work for that, if it doesn’t, build you own.

Setup Electron on Ubuntu

Start from a fresh ubuntu server.

$ docker-machine start dev
$ eval "$(docker-machine env dev)"

Running docker in interactive mode

# run docker in interactive mode
$ docker run -i -t ubuntu:14.04.3 /bin/bash

Install nodejs on Ubuntu

# install node
$ apt-get install -y curl
$ curl -sL https://deb.nodesource.com/setup_4.x | sudo -E bash -
$ sudo apt-get install -y nodejs

Install tape-run.

# install tape-run
npm i -g tape-run

Checking what’s missing

# checking missing dependencies
[email protected]:/# /usr/lib/node_modules/tape-run/node_modules/browser-run/node_modules/electron-stream/node_modules/electron-prebuilt/dist/electron --help
/usr/lib/node_modules/tape-run/node_modules/browser-run/node_modules/electron-stream/node_modules/electron-prebuilt/dist/electron: error while loading shared libraries: libgtk-x11-2.0.so.0: cannot open shared object file: No such file or directory

Install missing dependencies

# install missing dependencies
apt-get install -y libgtk2.0-0 libnotify-bin libgconf-2-4 libnss3 xvfb

Start xvfb server

# start xvfb server
export DISPLAY=':99.0'
Xvfb :99 -screen 0 1024x768x24 > /dev/null 2>&1 &

Start tape-run

# start tape-run
$ [email protected]:/# echo "console.log('yo'); window.close()" | tape-run
yo