Tuesday, April 1, 2014

Semi-Synchronous Replication at Facebook

 After intensive testing and hack, we started using Semi-Synchronous MySQL Replication at Facebook production environments. Semi-Synchronous Replication itself was ready since MySQL 5.5 (GA was released 3.5 years ago!), but I'm pretty sure not many people have used in production so far. Here are summary of our objective, enhancements and usage patterns. If you want to hear more in depth, please feel free to ask me at Percona Live this week.

Objective / Why Semisync?

  The objective of the Semi-Synchronous Replication is simple -- Master Failover without data loss, without full durability.

 First, let me describe why the objective is difficult without semisync.

 There are a couple of fast slave promotion (master failover) solutions. My own MHA covers both fully automated and semi-automated MySQL failover solution. Fully automated means both failure detection and slave promotion are done automatically. Semi automated means failure detection is not done but slave promotion is done by one command. Time to detect failure is approximately 10 seconds, and actual failover is taking around 5 to 20 seconds, depending on what you are doing during failover (i.e. forcing power off of the crashed master will take at least a few seconds). Total downtime can be less than 30 seconds, if failover works correctly. I'm using term "Fast Failover" in this post, which includes both automated and semi-automated master failover.
 In MySQL 5.6, GTID based failover is also possible. Oracle's official tool mysqlfailover automates MySQL master failover using GTID. The latest version of MHA also supports GTID.

 Both mysqlfailover and MHA rely on MySQL replication. MySQL replication is asynchronous. So there is a very serious disadvantage -- potential data loss risk on master failover. If you use normal MySQL replication and do automated master failover with MHA/mysqlfailover, you can do failover quickly (a few seconds with MHA), but you always have risks of losing recently committed data.

 If you don't want to take any risk of losing data, you can't do fast master failover with normal MySQL replication. You have to do the following steps in case of master failure.

- Always set fully durable settings on master. By fully durable I mean setting innodb_flush_log_at_trx_commit=1 and sync_binlog=1.
- On master crash, wait for a while (10~30 minutes) until the crashed master recovers. Recovery takes long time because it involves OS reboot, storage and filesystem recovery, and InnoDB crash recovery.
- If the crashed master recovers, you can continue services without losing any data. Since all data exist on the master, slaves can continue replication. BTW official 5.6 had a bug causing all slaves broken in this scenario, but this bug was fixed in 5.6.17.
- If the crashed master doesn't recover (H/W failure etc), you need to promote one of slaves to a new master. There is a risk of losing some data but you don't have any other choice.

 This "safer" approach has two issues.
- Longer downtime. This is because you have to wait for master's recovery.
- You can't eliminate risks of losing data. If master is dead and never recovers, your risk of losing data is the same as doing fast failover.

 So, in bad cases, you have to suffer from both longer down time and losing data.

 Semi-Synchronous Replication is helpful to prevent from losing data.

 If you do not care about data loss risk, there is no reason to use Semi-Synchronous replication. You can use normal MySQL replication and do fast failover with mysqlfailover or MHA. Facebook is one of the companies to care about data loss risk with MySQL, so that's why we were interested in Semi-Synchronous replication a lot.

 Semisync replication was originated from Google in 2007. Official MySQL supported from 5.5. Actual implementation algorithm was substantially different from Google's.

 MySQL Cluster and Galera offers synchronous replication protocol in different ways. I do not cover them in this blog post.

 Semi-Synchronous Replication currently has two types of different algorithms -- Normal Semisync and Loss-Less Semisync. Let me explain the differences.

Differences between Normal Semisync and Loss-Less Semisync

 Loss-Less Semisync is a new Semisync feature supported in official MySQL 5.7. Original implementation was done by Zhou Zhenxing as "Enhanced Semisync" project, and also filed as a bug report. Oracle implemented based on his idea, and named Loss-Less semisync for it. So Enhanced Semisync and Loss-Less Semisync have same meanings. I say Loss-Less semisync in this post.

 Normal semisync and loss-less semisync work as below.

1. binlog prepare (doing nothing)
2. innodb prepare (fsync)
3. binlog commit (writing to fscache)
4. binlog commit (fsync)
5. loss-less semisync wait (AFTER_SYNC)
6. innodb commit (releasing row locks, changes are visible to other users)
7. normal semisync wait (AFTER_COMMIT)

 On normal semisync(AFTER_COMMIT), committing to InnoDB is done before waiting for ack from semisync slave, so the committed rows are visible from applications, even though semisync slaves may not have received the data. If master is crashed and none of the slaves received the data, the data is lost but applications may have seen them. This is called phantom reads, and in many cases it's problematic.

 Loss-less semisync (AFTER_SYNC) avoids the problem. Loss-less semisync commits InnoDB after getting ack from one of semisync slaves. So when committed data is visible from applications, one of the semisync slaves have received that. Phantom read risk is much smaller: if both master and the latest semisync slave are down at the same time, data is lost. But it's much less likely to happen compared to normal semisync.

 To avoid data loss and phantom reads, Normal Semisync can't meet your expectations. Using Loss-Less Semisync is needed.
 With Loss-Less Semi-Synchronous replication, committed data should be on one of the slaves, so you can recover from the latest slave. You can always do fast failover here.

Reduced Durability

 When you do fast failover, you can set reduced durable settings on master as well as slaves. Reduced durability means innodb_flush_log_at_trx_commit != 1 and sync_binlog != 1. With Semi-Synchronous replication, you can immediately start failover when master is down. When promoting a slave to the new master, identify the latest slave (highly likely one of the Semi-Synchronous slaves but not guaranteed) and apply differential logs to the new master. Master's durability does not matter here, because there is no way to access master's data during failover. So you can safely reduce durability. Reducing durability has a lot of benefits.
- Reducing latency on (group) commit because it doesn't wait for fsync().
- Reducing IOPS because the number of fsync() calls is significantly reduced: from every commit to every second. Overall disk workloads can be reduced. This is especially helpful if you can't rely on battery/flash backed write cache.
- Reducing write amplification. Write volume can be reduced a lot, even less than half in some cases. This is important especially when using flash devices, because less write volume increases flash life expectancy.

Requirements for Semisync Deployment

 To make Semisync work, you need at least one semisync reader (slave with semisync enabled) within the same (or very close) datacenter as the master. This is for latency. When semisync is enabled, round-trip time(RTT) between master and one of the semisync slaves is added to transaction commit latency. If none of the semisync slave is located within close datacenter, RTT many take tens or hundreds of milliseconds, which means you can commit only 10~100 times from single client. For most environments, this will not work. You need a slave within close datacenter.

 To make fast failover work without data loss, you need to make sure Semi-Synchronous Replication is always enabled. MySQL Semisync has a couple of points where optionally semisync is disabled:
- Exceeding timeout (exceeding rpl_semi_sync_master_timeout milliseconds to get ACK from all of the semisync slaves)
- No semisync slave (can be controlled via rpl_semi_sync_master_wait_no_slave)
- Executing SET GLOBAL rpl_semi_sync_master_enabled=0

 If you want to enable semisync always, you make sure these scenario won't happen. Set infinite or very long timeout, and have at least two semisync readers.
 

 Facebook Enhancements to Semi-Synchronous Replication


 We spent a lot of time for testing Semi-Synchronous replication in 2013. We found some S1 bugs, serious performance problems, and some administration issues. Our MySQL Engineering team and Performance team worked for fixing issues and finally our Operations team deployed Semisync in production.

 Here are our major enhancements.

Backporting Loss-Less Semisync from 5.7

 As described above, Loss-Less Semisync is needed to prevent data loss and phantom reads, so we backported Loss-Less Semisync patch from official MySQL 5.7 to our Facebook MySQL 5.6 branch. It will be merged to WebScaleSQL branch soon.

 Interestingly, when we tested semisync performance, Loss-less semisync gave better throughput than normal semisync, especially when the number of clients is large. Normal semisync caused more mutex contentions, which was alleviated with loss-less semisync. Since Loss-less semisync has better data protection mechanism, we concluded there is no reason to use normal semisync here.

Semisync mysqlbinlog

 Starting from MySQL 5.6, mysqlbinlog supported remote binlog backups, by using --raw and --read-from-remote-server. On remote binlog backups, mysqlbinlog works like a MySQL slave. mysqlbinlog connects to a master, executing BINLOG DUMP command, then receiving binlog events via MySQL replication protocol. This is useful when you want to take backups of the master's binary logs. Slave's relay logs and binary logs are not identical to master's binary logs, so they can't directly be used as backups of the master's binary logs.

 We extended mysqlbinlog to speak Semisync protocol. The reason of the enhancement is that we wanted to use "semisync mysqlbinlog" as a replacement of local semisync slaves. We usually run slaves on remote datacenters, and we don't always need local slaves to serve read requests / redundancy. On the other hand, as described at above "Requirements for Semisync Deployment" section, in practice at least two local semisync readers are needed to make semisync work. We didn't like to run additional two dedicated slaves per master just for semisync. So we invented semisync mysqlbinlog and use it instead of semisync slaves, as shown in the below figure.




 Compared to semisync slave, semisync mysqlbinlog has a lot of efficiency wins.

- semisync slave has lots of CPU overheads such as query parsing, making optimizer plans. semisync mysqlbinlog does not have such overhead.
- semisync slave writes 2x (relay log and binary log). semisync mysqlbinlog writes binary log only.
- For semisync slave, the way to write to relay log is not efficient. IO thread writes to kernel buffer per each binlog event. For regular auto-committed transactions, it consists of three binlog events (query BEGIN, query body, and commit XID). When using InnoDB only, writing to kernel buffer for every XID event is enough (though it does not cover DDL). By writing to kernel buffer for every XID event, it makes the frequency of kernel buf flush by less than 1/3. semisync mysqlbinlog could easily do such optimizations. We have not done yet, but it is even possible to make mysqlbinlog send back ACK before writing, to a file, and the extension is very easy.
-  Slave causes contention between SQL thread and I/O thread, so IO thread itself slows down, which slows down semisync master throughput too. Semisync binlog does not have such overhead because there is no SQL thread.

 With mysqlbinlog reader, master failover step becomes a bit tricky. This is because mysqlbinlog is not mysqld process so it doesn't accept any MySQL command, such as CHANGE MASTER. When doing master failover, it is highly likely that one of local mysqlbinlog has the latest binary log events, and the events should be applied to a new master. New MHA version (0.56) supported the feature.

 In this configuration, mysqlbinlog processes need to be highly available. If all semisync mysqlbinlog processes are down, semisync is stopped or suffering from long wait time..


Reducing plugin_lock mutex contention

  Prior to MySQL 5.6.17, there was a performance bug that transaction commit throughput dropped significantly when there were non-semisync many slaves or binlog readers, even if there was only a few semisync readers. On typical deployments, there are two or three semisync readers and multiple non-semisync readers, so performance drop with many non-semisync readers was annoying.
 The performance drop was caused by "plugin_lock" MySQL internal mutex on master. For those who don't know, semisync is a plugin in MySQL, and it's not installed by default. The plugin_lock mutex was needed by semisync binlog dump threads only, but actually the mutex was held by all binlog dump threads. We looked into the problem further.
 First we tried replacing plugin_lock mutex with read/write mutex. It actually did not help much. But Linux profiling tools showed that plugin_lock still caused contentions. During profiling, we learned that most/all glibc rw-locks had an internal lock (mutex-like thing) on which threads could stall. The pattern was get lock, get exclusive access to cache line to modify data, release lock. This was relatively expensive for plugin_lock mutex, since it doesn't do any expensive I/O inside.

 So switching plugin_lock to read/write lock was actually a bad idea. It was needed to remove below plugin related locks as long as possible. There are four major plugin related mutexes in MySQL.
- plugin_lock
- plugin_lock_list
- plugin_unlock
- plugin_unlock_list

 We also noticed that Delegate classes had read/write locks and they caused very hot contentions (especially Binlog_transmit_delegate::lock). The read/write lock protects a list, so probably switching to lock-free list was possible. BTW we noticed that performance schema did not collect mutex statistics on the mutexes on Delegate classes (bug#70577).

 The real problem was all of the above locks were held not only by semisync binlog readers, but also non-semisync binlog readers.

 Based on the above factors, we concluded removing all plugin mutexes was not easy, then we decided to optimize to hold these locks by semisync binlog readers only, and not holding by non-semisync binlog readers. The below is a benchmark result.



 x-axis was the number of non-semisync binlog readers, y-axis was concurrent INSERT throughput from 100 clients. The number of semisync binlog readers was always 1 to 3. Detailed benchmark conditions were described in a bug report.
 Hopefully our patches were finally merged to 5.6.17 and 5.7 so everybody can get benefits easily.


 With all of the enhancements, we could get pretty good benchmark results with semisync.


 This is a mysqlslap insert benchmark on the master, with one semisync slave/mysqlbinlog running. x-axis is the number of clients, y-axis is the number of inserts on the master. Enhanced means loss-less semisync.
 Normal slave is traditional (non-semisync) slave. Enhanced mysqlbinlog is our semisync usage pattern. As you can see, loss-less semisync beats normal semisync due to internal mutex contention reductions. semisync mysqlbinlog also beats semisync slave because of much less overheads. This shows that loss-less semisync scales pretty well.
 

Conclusion and Future Plans

 After several performance improvements, Semi-Synchronous replication became good enough for us. From performance point of view, I expect that single-threaded application performance will be next low-hanging fruits. On our benchmarks, we got around ~2500 transaction commits per second with semisync (0.4ms per commit). Without semisync, it was easy to get ~10000 transaction commits  per second (0.1ms per commit). Of course semisync adds RTT overhead, but on local datacenter network, RTT is much lower than 0.3ms. I think there is another semisync overhead here, so will revisit this issue and will work with Oracle Replication team and outside experts.


660 comments:

«Oldest   ‹Older   601 – 660 of 660
ALAN JOHN said...


chloroplast diagram
where does the calvin cycle take place
rough endoplasmic reticulum definition

Shubham Meena said...

Happy Raksha Bandhan 2017
Happy Raksha Bandhan Images
Happy Raksha Bandhan SMS
Happy Raksha Bandhan Images HD
Raksha Bandhan Whatsapp DP
Raksha Bandhan Whatsapp Status
Happy Raksha Bandhan Quotes
Happy Rakhi Images
Raksha Bandhan Photos
Happy Raksha Bandhan Messages
Raksha Bandhan Shayari
Raksha Bandhan Thali Decoration Designs

Ravi S said...

Really fantastic artical thanks for sharing


https://zinnga.com/collections/buy-lehengas-online

Pooja Rao said...

Delhi Education Department Recruitment 2017-18
Delhi Post Office New Exam Date 2017
SSC CHSL Syllabus 2017
Osmania University Admission 2017

RdxPunjaB said...

happy raksha bandhan images 2017

happy diwali 2017

happy raksha bandhan images

ishan patel said...

Cheap web hosting in india

Hotshot Hassan said...


india independence day
when does independent assortment occur
avast secureline vpn license

Unknown said...

Images for Friendship Day
Friendship Day Images Download
Download Friendship Day Images
Pics of Friendship Day

ishan patel said...

Internet explorer for android
free download internet explorer for android
download internet explorer for android
download internet explorer for android phones
internet explorer for android download
internet explorer 10 for android
internet explorer 11 for android
internet explorer download for android

Saini Ratanlal said...

Images for Friendship Day
Friendship Day Images Download
Download Friendship Day Images
Pics of Friendship Day

Rajkumar Bagri said...

Friendship Day 2017 Images
Happy Friendship Day Pictures
Happy Friendship Day Pics
Friendship Day SMS
Friendship Day Messages

Akash Sharma said...

notepad tricks
popads review
popads reviews 2017
Download photoshop cs6 with key
idm fake fixer software download
idm fake serial key
whatsapp tips and tricks
whatsapp tricks
how to access someone whatsapp
sharmatricks

Akash Sharma said...

notepad tricks
popads review
popads reviews 2017
Download photoshop cs6 with key
idm fake fixer software download
idm fake serial key
whatsapp tips and tricks
whatsapp tricks
how to access someone whatsapp
sharmatricks

John Cena said...

https://wwesummerslams.com/

yogareddy is said...

The most unique way to celebrate Friendship Day would be hold a gathering for your favorate images,quotes,
hd wallpapers,messages,pics,photos and many more... Be that as it may, this thought needs a touch of arranging and exertion as you have to choose where to post(facebook,whatssapp,twitter,instagrame etc..) Once that is done, ring your pals and simply have your indulgence !

happy friendship day images

happy friendship day 2017

friendship day quotes

happy friendship day quotes

friendship images with quotes

friendship day messages

friendship day wishes

happy friendship day message

friendship day status
friendship day date

happy friendship day date

happy friendship day images

An extraordinary thought to praise friendship in an inventive way is make a blurb on Friendship Day this year.

friendship day 2017

friendship images

friendship day pic

friendship day pics

friendship day photos
For some the best sort of Friendship Day festivity is maybe a heart-to-heart converse with a best friend Sharing everything that has made a difference to you and your frinds for so......

friendship day wallpapers

photos of friendship

friendship quotes images



Ashok Kumar said...

Images on Friendship Day
Happy Friendship Day 2017 Images
Friendship Day Photo Download
Friendship Day Wallpaper Download

kongponleu88 said...

Thanks for this informative article, I hope you will get most positive response specially for this post. . . .
โกเด้นสล็อต

priyanshu beautique said...

Through the passage of time festivals are undergoing modifications. Raksha Bandhan is also known as Rakhi. Rakhi has become a sacred festival for sisters and brothers. Sisters tie them to brothers. Priests tie them to people of his congregation.For More..
Plz visit:- Rakhi Wishes

priyanshu beautique said...

India is a country of colorful festivals. All the festivals in India are according to the Indian Hindu Calendar.Each one of them has either religious significance or they signify different Indian relations.For More..
Plz visit:- Raksha Bandhan Images

priya beautique said...

Rakhi is the traditional Indian festival where a sister ties Rakhi string around her brother's wrist.
Like many other Indian festivals, this too is a gift-giving occasion when brother and sisters exchange their token of love.
There are many quotes are available for sibling in our article......
For More....
Plz visit:- Tomorrowland 2017 Live

سازه برتر said...


پروژه عمران منبع پروژه های دانشجویی عمران و معماری آماده برای ارائه توسط دانشجویان عمران ، ساختمان و معماری. فایل های مورد نیاز خود را از سایت پروژه عمران دانلود کنید:
پروژه معماری
پروژه عمران
پاورپوینت معماری
پاورپورینت عمران



فروش فایل
دانلود فایل
دانلود پروژه
دانلود پاورپوینت
دانلود مقاله

Kathy Smith said...

I have read your post, it was very helpful to me.
Thank you for share the post.
see this here

Mai Cyrril said...

Danke, dafur dass Ihr jeden Tag mit uns die Hl. Messe gefeiert habt, fur all die Reparaturen im ganzen Haus Jeux De Friv Juegos Friv Jogos Friv fur Euere Bereitschaft Juegos Friv Jogos Friv Juegos Friv immer zur Verfugung zu stehen Juegos Geometry Dash Juegos Twizl Twizy Juegos Twizl Danke, dass Sie Ihr Muhen um den Aufbau des Leibes Christi mit uns teilten.

ojas said...

Raksha Bandhan 2017 Whatsapp Funny Videos for Boys :-Hello friends welcome in here. Use these all Raksha Bandhan Whatsapp Funny Videos and Raksha Bandhan 2017 Whatsapp Videos for Boys. As you, all know about the Raksha Bandhan.

Raksha Bandhan 2017 Whatsapp Funny Videos for Boys

Raksha Bandhan is a festival that is celebrated all over the India. Raksha Bandhan is the festival of brother and sister. In the occasion of Raksha Bandhan sister tie a beautiful rakhi on their brother’s hand. As you, all know the meaning of Raksha Bandhan.



Download Free Rakhi Funny Videos 2017

Raksha Bandhan is a bond of protection. This is a festival when brother and sister celebrate this festival with each other. This is a very wonderful relationship between brother and sister. In this day sister pray for his brother long and safety life. In return, brother gives their sister a best gift and surprise on the occasion of Raksha Bandhan. Enjoy these all Raksha Bandhan funny Videos.

News Media said...

Jab Harry Met Sehjal

Jab Harry Met Sehjal Full Movie

Download


Download Jab Harry Met

Sehjal


Jab Harry Met Sehjal Watch Online



Jab Harry Met Sehjal Mp4 Download



Angel Arora said...

Jab Harry Met Sejal 2017 Full Movie Downlaod
Jab Harry Met Sejal SRK 2017 Full Movie Downlaod
Download Jab Harry Met Sejal 2017 Full Movie
JHMS SRK 2017 Full Movie Downlaod

Jab Harry Met Sejal Full Movie Mp4 Download
Download Jab Harry Met Sejal 2017 SRK Movie In Mp4
JHMS 2017 SRK Movie In Mp4


Jab Harry Met Sejal 2017 Watch online
JHMS 2017 SRK Watch online


Download 720p Jab Harry Met Sejal
JHMS 2017 SRK 720p Download

قمة الدقة said...

قمة الدقة
شركة نقل اثاث بالرياض
شركة نقل عفش بالرياض
شركة تنظيف خزانات بالدمام
شركة تنظيف خزانات بالخبر
شركة عزل اسطح بالدمام
شركة عزل اسطح بالخبر

priyanshu beautique said...

Janmashtami is one of the India's most popular festival celebrated across the country with great zest and zeal,to commemorate the birth of Lord Krishna. Considered one of the most powerful human incarnations of Lord.for more..
Plz visit:- Janmashtami
Shri Krishna Janmashtami is the Hindu festival celebrated every year on Krishna paksh ashtami of Bhadrapada month according to the Hindu calendar. The day is also known as gokulashtami, saatam aatham, rohiniashtami etc. for more...
Plz visit:- Janmashtami 2017

robin singh said...


Happy New Year 2018 Images
Happy New Year Images 2018
Black Friday 2017
Jio Mobile
Jio Mobile Online
SBI Recruitment
Govt Jobs 2017
Sarkari Naukri
Post Office Recruitment
Latest Recruitment

Happy New Year 2018 Images
Happy New Year Images 2018
Black Friday 2017
Jio Mobile
Jio Mobile Online
SBI Recruitment
Govt Jobs 2017
Sarkari Naukri
Post Office Recruitment
Latest Recruitment

robin singh said...

15 August 2017 Slogans
15 August 2017 SMS
15 August 2017 Wishing Quotes
15 August 2017 Profile Pics
15 August 2017 Animated Pics
15 August 2017 Greetings
15 August 2017 Quotes
15 August 2017 Message
Labor Day Images
Labor Day Slogans

robin singh said...

Labor Day Message For Employee
Labor Day Songs
Labor Day SMS
Labor Day Poems
Labor Day Wishing Quotes
Labor Day Greeting Cards
Happy Labor Day 2017
Labor Day Wishes
Labor Day Cliparts
Labor Day Images

robin singh said...

Janmasthami 2017 Wishes
Janmasthami 2017 Greetings
Janmasthami 2017 Wishes in Hindi
Janmasthami 2017 Poems
Janmasthami 2017 Wallpapers
Janmasthami 2017 Wishing Quotes
Janmasthami 2017 Pictures
Janmasthami 2017 Wishes
Indian Air Force Recruitment
Uttarakhand TET Application 2017

robin singh said...

Janmasthami 2017 Wishes
Janmasthami 2017 Greetings
Janmasthami 2017 Wishes in Hindi
Janmasthami 2017 Poems
Janmasthami 2017 Wallpapers
Janmasthami 2017 Wishing Quotes
Janmasthami 2017 Pictures
Janmasthami 2017 Wishes
Indian Air Force Recruitment
Uttarakhand TET Application 2017

Angel Arora said...

Toilet Ek Prem Katha 2017 Full Movie Downlaod
Toilet Ek Prem Katha SRK 2017 Full Movie Downlaod
Download Toilet Ek Prem Katha 2017 Full Movie
Toilet Ek Prem Katha Full Movie Mp4 Download
Download Toilet Ek Prem Katha 2017 Movie In Mp4
Toilet Ek Prem Katha 2017 Watch online
Download 720p Toilet Ek Prem Katha

sandhosh said...

Thank you for this great article.keep more updates.
Digital marketing company in Chennai

Amazing World said...

krogerfeedback

Priya Rajendran said...

Great and interesting article to read..

best big data training in chennai

Anshul Agarwal said...

clash of clans mod apk download
clash of clans modded apk
clash of clans unlimited gems apk
clash of clans unlimited gems apk no root

event blogging said...

Festival Celebration 2017 | Pictures | Photos | Images | Wishes | Wallpapers | Message |

Happy Ganesh Chaturthi Happy Ganesh Chaturthi Images and Wallpapers., Photos, Wallpapers, GIF’s and Pics for Facebook and Whatsapp DP

Why Ganesh Chaturthi Celebrated?

Happy Ganesh Chaturthi 2017, Happy Ganesh Chaturthi Images and Wallpapers., Why Ganesh Chaturthi Celebrated?

Ganesha Chaturthi images and wallpaper 2017 [Free HD]

Ganesh Visarjan 2017 Images - HD images of Dance, Celebration, Visarjan Process

GATE 2017 said...

Happy Diwali GIF for WhatsApp

Happy Diwali Quotes 2017

Happy Diwali Wishes in Hindi

Happy Diwali SMS Messages in English

Happy Deepavali Msgs in Hindi

Happy Diwali Greetings Cards Free

Atul Saini said...

if u searching the massage service massage singapore

priyanshu beautique said...

You can go for a change from the conventional salwar suits to various gorgeous designer kurti. Designer kurtis are emerging with fashion and style which not only gives you a different flavour but also adds to your wardrobe and beauty.for more..
Plz visit:- Kurti
The designer kurtis are certainly the most mesmerizing creation of the Indian designers. These are basically the modified version of the traditional ladies kameez.for more..
Plz visit:- Designer Kurti

Atul Saini said...
This comment has been removed by the author.
Atul Saini said...

if you searching for dog boarding service click here. for reviews then decide dog boarding singapore review

M EL3raBY said...

نقدم لكم بإستمرار مجموعة رائعة من العاب تلبيس بنات وأيضا يحتوي الموقع علي تشكيلة ممتعة العاب بنات ستايل وما يميز موقعنا أن جميع الألعاب متاحة مجانية تلعب مباشرة اونلاين بدون تحميل علي الكمبيوتر, نقدم لكم بإستمرار مجموعة العاب مجانية حلوة علي موقعنا كل يوم تجدون العاب فلاش حصرية مثل العاب تلبيس بنات التي يحبوها جميع الفتيات الجميلات فهي من أمتع الألعاب الموجودة علي الإنترنت الذي يهتم بتقديم أحسن محتوي عربي لألعاب البنات المتنوعة, كما أن العاب بنات ستايل عبارة عن موقع ترفيهي يقدم لكم مجموعة كبيرة جدا ومختلفة من أجدد العاب الطبخ والعاب قص الشعر وكذلك العاب مكياج وتلبيس ويمكنكم جميعا الآن الإستمتاع في لعب العديد من الالعاب المتميزة والحلوة فقط علي موقع العاب كركر .

Nikshit Narula said...

Awesome Post! Let us know more about Happy Diwali 2017 Whatsapp Status
Also download latest movies Stranger Things
Also Read trending Stories Tragically Selfie Attempt
Also Check out Latest vacancy of Government Jobs Indian Army Recruitment 2017

Nalin Nishant said...


fb page liker
auto follower fb
page liker
facebook page liker
fb auto follower

Nalin Nishant said...

Nice admin
fb page liker
auto follower fb
page liker
facebook page liker
fb auto follower

ishan patel said...

Hello visit our website for best Cargo TrailerWe are providingenclosed trailerand must visit website forenclosed motorcycle trailers.we are also providing following trailers that includes
car haulers
Diamond cargo
cheap enclosed trailer
Concession trailers
Enclosed Gooseneck Trailers
Transport trailers
It is one of the best website for trailer.We hope you will like it.

Atul Saini said...
This comment has been removed by the author.
Atul Saini said...

If you searching best massage service in singapore
tantric massage

ishan patel said...

Hello visit our website for best Cargo Trailer We are providing enclosed trailer and must visit website for enclosed motorcycle trailers.we are also providing following trailers that includes
car haulers
Diamond cargo
cheap enclosed trailer
Concession trailers
Enclosed Gooseneck Trailers
Transport trailers
It is one of the best website for trailer.We hope you will like it.

Hút bể phốt thông tắc cống ở Hà Nội said...

hút bể phốt
hút bể phốt ở hà nội
hút bể phốt tại hà nội
hút bể phốt giá rẻ tại hà nội
thông tắc cống
thông tắc cống ở hà nội
thông tắc cống tại hà nội
thông tắc cống hà nội
thông tắc cống giá rẻ tại hà nội

Thông tắc cống hút bể phốt ở tại Hà Nội giá rẻ said...

hút bể phốt giá rẻ ở tại Hà Nội http://thongtaccong-hutbephot-giarenhat.com/

Nikshit Narula said...

Awesome Post
Check out Latest Government Job of UKMSSB Recruitment 2017

Thu Lê said...

I was very impressed by this post, this site has always been pleasant news. Thank you very much for such an interesting post. Keep working, great job! In my free time, I like play game: douchebagworkout2.org. What about you?

Dream11 Admin said...

pocket morty Recipes
Pocket Morty Craftig Recipes
Pocket morty Recipe
Pocket morty Recipe
pocket mortys Recipes
Pocket morty Recipe
pocket mortys Recipes
pocket mortys Recipes

Dream11 Admin said...

pocket mortys Recipe
pocket mortys Recipe
pocket mortys Recipe
pocket mortys Recipe
pocket mortys Recipe
pocket mortys Recipe
pocket mortys Recipe
pocket mortys Recipe

Dream11 Admin said...

pocket morty recipes
Pocket Mortys Recipes
Pocket Mortys Recipes
pocket mortys crafting recipes

Dream11 Admin said...

pocket morty recipes
Pocket Mortys Recipes
Pocket Mortys Recipes
pocket mortys crafting recipes
Pocket Mortys Recipes
pocket mortys crafting recipes
Pocket Mortys Recipes
Pocket Mortys Recipes

«Oldest ‹Older   601 – 660 of 660   Newer› Newest»

Post a Comment