Tuesday, 23 August 2016

Magento Enterprise Edition 2.1- Experience Commerce without Limits

Magento 2 is transforming the commerce field dramatically with the new experience it offers to those who rely on it. Due to the new powerful features of the next generation platform, marketers and merchants are able to do more and reach more as a team.

Magento Enterprise Edition 2.1 clears all the boundaries and gives you to create an unique shopping experience and takes your business to next level.  Built for rapid and cost-effective innovation, the platform combines rich, out-of-the-box functionality, enterprise performance and scale, and powerful business tools to keep you ahead of increasingly complex commerce operations and growing customer demands.

Features Highlights of Magento EE 2.1
Magento EE 2.1 has more exiting features which helps to speed up the performance and reduces development and deployment efforts with more customization flexibility.

Powerful Admin Experience
Work efficiently with a modern and intuitive user interface. A dashboard lets you monitor your business with quick access to sales, order, search term and best-selling product data. Enhanced navigation and search makes it easy to find information, and you can configure and save different views of order and customer management screens to speed up day-to-day tasks.

Responsive Design Themes
Provides built in touch-friendly and easily customizable with SEO features.  The responsive themes adapt and adjust how menus, images, checkout and other features are displayed to fit desktop, laptop, tablet, and mobile device screens.

Customer Segmentation and Personalization
Create personalized shopping experiences that boost conversion rates by dynamically displaying content, promotions, and pricing to specific customers based on properties such as their location, gender, order history, lifetime purchase value, wish list items, and more. You can also target unknown site visitors based on the products they have viewed or items in their cart.

Dynamic Rule-Based Product Relations
You can set your own automated rules to show-case up-sell, cross-sells and related products to each customer segments. Rules are easily administered through a condition-based tool allowing you to effortlessly target product suggestions to increase sales and average order values.

Content Staging and Preview
Your business team can easily mange wide range of contents without involving any developer. Preview all changes by date or store view to guarantee a flawless shopper experience and optimize the timing and impact of site updates by managing all changes through a timeline dashboard. Automatically deploy updates at scheduled times for greater efficiency.

Enhanced payment options
Enhanced payment options of Magento Enterprise Edition 2.1 boost conversion rates like never before! Customers can now make a purchase with PayPal in overlay window of the merchant's account without actually even leaving your store.

Manage huge catalogs and easily scale search capacity as queries grow with integrated Elasticsearch technology. Set-up is fast and global deployments are simple with support for 33 languages out of the box. Elasticsearch offers suggestions for customer misspellings, support for stop words and attribute weighting, and synonym management via the Magento Admin to increase relevancy and conversions.

 Visual Merchandising
Optimize product category pages to drive higher sales by arranging products with a simple drag-and-drop interface or sorting rules that order products by best seller, color, highest margin, or newest addition. Save time by setting rules to automatically assign products to specific categories based on their attributes, such as brand, price, or date created.

Multiple Master Databases
Provides provision to scale the database tier of Magento application by using seperate master database to support checkout, order Management and catalog data. This ensures merchandising and order management activities never impact the shopper’s ability to browse and make purchases on the website.

Flexible Deployment Options
Choose to deploy your Magento software in whatever environment suits your goals and operations best, including cloud, 3rd party hosting, and on premises. New Magento cloud service, Enterprise Cloud Edition, enables rapid deployment of fully customizable, secure, and scalable web storefronts on a redundant, AWS-based hosting and managed services infrastructure that is optimized for performance and resilience.

Varnish Page Caching
Speed up your site performance, decrease response time and reduce load on servers with integrated Varnish page caching. Magento fully supports and provides Varnish configuration files out-of-the-box to make it easy to implement.

Hope this article helps you to know some additional out-of-the-box features in  Magento Enterprise Edition 2.1.

Ideaapplied Technologies provides Magento eCommerce portal development service with complete project management suite. You can visit us at ideaapplied.com to know more about our services.

Salt-stack connect minion to a different master

This is a guide if you want to connect a minion from one master to a different master.


You have a minion ex. minion1 and it is currently connected to master1. Now for some reason you want to connect to a different master ex master2. I had this requirement when I had to create a clone of a live ec2 instances minions which is already connected to live-master and I had to connect it to a master which we use for development i.e. dev-master.

NOTE:  I am assuming the OS to be Ubuntu

The following are the steps.
  • SSH into minion and Shutdown salt-minion on the minion instance using sudo service salt-minion stop (Optional)
  • Restart master2 in open mode(This is temporary, This will make it easier) by doing the following
    • In master2 (/etc/salt/master) configuration set open_mode: True
    • Restart salt-master using sudo service salt-master restart
  • Change ID(Optional), master IP/HOST and master_finder of minion by doing the following
    • ssh to minion
    • do sudo vim /etc/salt/minion
    • search for id: and change it if needed.
    • search for master: IP/HOST, change the IP or HOST for the master2
    • search for master_finger: and delete the property. This is newly added so It may not be there in configuration.
    • Remove key for old master(master1) server rm /etc/salt/pki/minion/minion_master.pub
    • Restart minion sudo service salt-minion restart
  • Change open_mode to False or remove the property(by default is False) and restart master2.

This should allow the minion to connect to the master2. Now there are things which you would need to change depending on the requirement. Like I was creating a different environment by cloning liveand we have a grain named circle set for the environments like devstaging or live. So had to change the grain to livetest.
salt '*.livetest.example.com' grains.setval circle livetest
Hope it helps.

Any suggestions or questions are welcome.

Ideaapplied Technologies provides comprehensive QA services. Visit us at ideaapplied.com to know more about our services

Wednesday, 10 August 2016

Salt-stack mysql use root user from any host

I recently had a problem while installing mysql on a server using salt-stack.

The scenario is that there is a server where I install mysql and other servers should be able to access it. Now these are ec2 instances and the security group is setup such a way that only server having certain security group can access the mysql server. So even if I open mysql for any host, There would not be any security issues.

The problem was that I was not able to login to root user from any other server because the root user login was only allowed from the localhost if I do the following.

    - installed
    - pkgs:
      - mysql-server
      - python-mysqldb
    - running
    - name: mysql
    - enable: True
    - require:
      - pkg: mysql-server
    - watch:
      - file: /etc/mysql/my.cnf
    - present
    - name: root
    - password: {{ pillar['mysql']['server']['root_password'] }}
    - require:
      - service: mysql
So what I had to do for the servers to login as root user into mysql database from any host including localhost is to add another user.preset for root again with % as host.

    - name: root
    - host: '%'
    - password: {{ pillar['mysql']['server']['root_password'] }}
    - connection_user: root
    - connection_pass: {{ pillar['mysql']['server']['root_password'] }}
    - connection_charset: utf8
    - saltenv:
      - LC_ALL: "en_US.utf8"
    - require:
      - service: mysql
This works for any user which should be accessible from any host including localhost.
Any questions or suggestions are welcome.

Ideaapplied Technologies provides comprehensive QA services. Visit us at ideaapplied.com to know more about our services

Thursday, 4 August 2016

salt-cloud install a specific version of salt-minion

This is a guide to install a specific version of salt-minion package on new minions being bootstrapped using salt-cloud.
This is not specific for any OS or destro but I was using Ubuntu 14.04.
salt-cloud uses salt-bootstrap.sh script (https://github.com/saltstack/salt-bootstrap) which installs every dependencies and salt-minion on the newly created instance. This file is usually located at /usr/lib/python2.7/dist-packages/salt/cloud/deploy/bootstrap-salt.sh.
Version and/or Source can be specified in the cloud.profile configuration as below.
In /etc/salt/cloud.profiles.d/someprofile.conf

  provider: aws
  image: ami-xxxxxx
  size: m1.small
    - SG1
    - SG2
      circle: staging
      type: app
  script_args: git v2015.5.2

The last property script_args value will be passed to the bootstrap script. The above values tells to install salt-minion version v2015.5.2 from github(not the Ubuntu repo).
The values can be any of the following.
Installation types:

- stable (default, It installs from OS Repository e.g Deb/RPM)
- stable [version] (ubuntu specific)
- daily  (ubuntu specific)
- testing (redhat specific)
- git (Installs from github repository)


- stable
- stable 2015.5
- daily
- testing
- git
- git develop
- git v2015.5.3
- git 8c3fadf15ec183e5ce8c63739850d543617e4357

Custom/Edited script can also be used instead of the default bootstrap-salt.sh
Just copy the /usr/lib/python2.7/dist-packages/salt/cloud/deploy/bootstrap-salt.sh or download https://github.com/saltstack/salt-bootstrap/blob/develop/bootstrap-salt.sh and put it under/etc/salt/cloud.deploy.d/. If the file is named bootstrap-salt.sh then no change required in the profile configuration. If the file name is something different like custom-bootstrap.sh then added the following in the configuration file.
In /etc/salt/cloud.profiles.d/someprofile.conf

  provider: aws
  image: ami-xxxxxx
  size: m1.small
    - SG1
    - SG2
      circle: staging
      type: app
  script: custom-bootstrap.sh # This can also be an absolute path.
  script_args: git v2015.5.2

Hope it helps.
Any questions or suggestions are welcome.

Ideaapplied Technologies provides comprehensive QA services. Visit us at ideaapplied.com to know more about our services

Wednesday, 20 July 2016

c3p0 and MySQL, smart configuration.

I recently answered a question related to c3p0 and MySQL on stackoverflow. There are a lot of misconceptions about c3p0 properties in relation to database properties. I will try to explain them in detail. There is no code here so if you are looking for an integration code then you know what to do.

Let's first understand a few mysql properties which directly effects your application datasource configuration.

interactive_timeout : (Default 28800 sec) interactive_timeout is for mysql shell sessions in seconds like mysqldump or mysql command-line tools. Most of the time this is set to higher value because you don't want it to get disconnected while you are doing something on mysql cli. This does not really effect your app config but to avoid confusion with regards to wait_timeout.

wait_timeout : (Default 28800 sec, AWS RDS 300 sec) The amount of seconds of inactivity that MySQL will wait before it closes a connection on a non-interactive connections. i.e connected from java application.

Now let's understand c3p0 properties in relation with the above DB properties.

maxIdleTime: (Default: 0) Seconds a Connection can remain pooled but unused before being discarded. Zero means idle connections never expire.
This refers to how long a connection object can be usable and will be available in pool. Once the timeout is over c3p0 will destroy or recycle it.

Now the problem comes when you have maxIdleTime higher than the wait_timeout. let's say if the mxIdleTime : 50 secs and wait_timeout : 40 sec then there is a chance that you will get Connection timeout exception: Broken Pipe if you try to do any operation in last 10 seconds. So maxIdelTime should always be less than wait_timeout.

Everything is easy up until now. Here comes the real problem.

Any of the data source properties(eg. maxIdleTime) only affects the connections which are in pool so if your application has acquired a connection, keeps it idle for more than maxIdleTime and then tries to do any operation, it will definitely get "Broken Pipe". It's like maxIdleTime is a warranty period of a connection.
You can reproduce this easily by following these steps. 1) set wait_timeout to 15 sec 2) Set maxIdleTime to 14sec 3) acquire a connection 4) Thread.sleep(16000). 5) run any query on the connection...  Saw that?

There is no ideal wait_timeout. You have figure it out yourself based on your application requirement. It is good to have higher wait_timeout on mysql but It's not always practical especially when you have an application already built.
If you are thinking of changing it then you have to make sure that in your application you are not keeping any connections idle for more that wait_timeout.

This is especially important when you are not doing connection management manually for example when you use Spring transnational API. Spring starts transaction when you enter an @Transaction annotated method so it acquires a connection from pool. If you are making any web service call or reading some file in the transactional method which takes more time than wait_time out then you are screwed.

I have faced this issue before. In one of the application there was a cron which would do order processing. To make it faster I used batch processing. Once a batch of customers is retrieved and the code does some processing(no db calls), And after that when It tried to save all the orders there was a "Broken Pipe". The problem was my wait_timeout was 1 minute and order processing was taking more than that. So I had to increase it to 2 minutes. I could have reduced the batch size but that was making the overall process slower.

You also have to consider that acquiring a connection is expensive task and if you have wait_timeout too low then it beats the whole purpose of having a connection pool, as it will frequently try to acquire new connections.

Ideal Timeout

There is no such thing as ideal timeout but here's some of the steps you can follow to better manage it.

  • When development is in progress set the maxIdleTime, and wait_timeout to as low as possible, this will make developers write better queries. If they don't, it will throw "Broken Pipe"
  • If you have long running transactions for example if you are running a cron to run some batch processes, then move those process to separate application.
  • If you are not doing transaction management manually in your application i.e spring transactional API. then make sure that the method marked transaction does not do anything which would take long time for example calling a webservice or reading a file etc.
  • Longer transaction would also lead to Deadlocks(Lock wait timeout) errors because if a transaction is in progress which is doing any updates then other transaction will wait and fail once the innodb_lock_wait_timeout is exceeded.

Ways to configure c3p0

  • Manually control wait_timeout
    • In this configuration you use maxIdleTime in the configuration and set it lesser than the wait_timeout that you have set in mysql.
    • This way you can be sure that the connection will be open longer than the wait_timeout and thus avoid broken pipe.
    • This method is quite simple, you have full control over the wait time. The major drawback is that it is db property agnostic i.e you application will fail if it not in sync with wait_timeout on db side.
  • Let c3p0 take care of connection life.
    • In this type you use many other c3p0 property except maxIdleTime as below.
    • testConnectionOnCheckin validates the connection when it is returned to the pool. testConnectionOnCheckOut, although would ensure active connections before use, would be too expensive to do.
    • idleConnectionTestPeriod sets a limit to how long a connection will stay idle before testing it. Without preferredTestQuery, the default is DatabaseMetaData.getTables() - which is database agnostic, and although a relatively expensive call, is probably fine for a relatively small database. If you're paranoid about performance use a query specific to your database (i.e. preferredTestQuery="SELECT 1")
    • maxIdleTimeExcessConnections will bring back the connection count back down to minPoolSize after a spike in activity.
    • This method works fine and does not depend on db configuration much but there is so much overheads, for example it will run an extra query every time the application is tries to get a connection, for a normal web application it is huge because if there are 1K user online any making 2K clicks every minute it will execute extra query every time, because normal application would acquire a lock and release it before sending a response.
Ideaapplied Technologies provides comprehensive QA services. Visit us at ideaapplied.com to know more about our services.