sorry I did not cover this today, so I will answer here:
- there are many approaches to the first question. Some are rigorous and follow from the high degree of symmetry of the matrix we consider, which is such that it has symmetric and antisymmetric eigenvectors. A more physics-like approach is that you ask questions like: should I expect the 1st and 4th atom to have different amplitudes given that their environment looks identical? (No). Same for atoms 2 and 3. Picturing the possible motions also helps. This leads to modes such as [1 a -a -1], or [1 b b 1]. Considering that the center of mass should remain fixed, we could have immediately guessed that b=-1 for the oscillating mode, whereas b=1 gives the translational mode of the entire molecule.
- to the 2nd question: the approach you suggest is good. Perhaps your question is on how to expand y=\sqrt{a^2\cos^2(x)+b^2} around x=\pi/2 for b\ll a . In this case, even though b \ll a, the first term becomes the smallest when x\rightarrow\pi/2. Therefore, expanding the square root around x=\pi/2 gives y\approx b(1+\frac{1}{2}\frac{a^2}{b^2}\cos^2(x)). In contrast, expanding around x=0 gives y\approx \sqrt{a^2 \cos^2(x)}
If, on the other hand, b \gg a, the 2nd term is always larger than the first, so we get y\approx b(1+\frac{a^2}{b^2}\cos(x)) for all x